Self-Supervised Visual Representation Learning

Follow this topic Ask about this topic

Momentum

20 papers in the last four weeks, up 100% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 151

Aug 31, 2026cs.CV

Motion-Saliency Complementary Masked Modeling for Point Cloud Video Understanding

Point cloud video representation learning is crucial for 3D dynamic scene understanding. In this paper, we propose MoSaiC, a novel Motion-Saliency Complementary masked modeling framework for self-supervised point cloud video representation learning. MoSaiC couples three components: Curriculum Motion-Saliency Masking (CMSM), which guides the masking process toward motion-salient tokens under a curriculum schedule; Normal-Flow Motion (NFM) modeling, which supervises the local rigid rotation of each token in the Lie algebra so(3) as an explicit geometric motion target; and Cross-view Token Consistency Prediction (CTCP), which enforces consistency between two complementary masked views at the token level. Together, these components allow MoSaiC to effectively capture both appearance and motion dynamics. Extensive experiments on multiple downstream tasks, including action recognition, temporal action segmentation, and point-level semantic segmentation, demonstrate the effectiveness of our approach.

Wei Wang, Yiding Sun, Yuyan Wang +4

School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, 710049, China · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), Shenzhen, 518107, China · School of Software Engineering, Xi’an Jiaotong University, Xi’an, 710049, China +4

Point Cloud Learning Generative Modeling Motion Representation Learning Self-Supervised Visual Representation Learning

Aug 27, 2026cs.CV

DINOcular: Self-Supervised Visuospatial Representations

We introduce a self-supervised framework for learning joint visuospatial representations from RGB-D observations. While modern vision foundation models are trained almost exclusively on RGB images, many embodied systems have access to explicit depth sensing, which provides geometric information that monocular inputs cannot recover. Our method integrates depth-derived geometric priors with a visual backbone through inter-patch and intra-patch fusion, enabling the model to encode both appearance and spatial structure efficiently. The resulting representation shows promising improvements on 3D awareness while preserving semantic transfer: it outperforms prior methods of comparable scale on multiple 3D geometry benchmarks, and remains competitive when probed for standard RGB-D semantic segmentation tasks.

Farkhat Almukhamedov, Sami Azirar, Hermann Blum

Robot Perception and Learning Lab University of Bonn & Lamarr Institute

Visual Representation Learning Geometric Representation Learning 3D Representation Learning Self-Supervised Visual Representation Learning

Aug 20, 2026cs.CV

RIPE++: Reinforced Keypoint Learning from Positive Pairs Only

Sparse keypoint extraction and matching underpin core tasks in geometric computer vision, including structure-from-motion, visual SLAM, augmented reality, and medical image registration. Learning robust local feature representations, however, typically requires accurate camera poses or depth supervision, which are often unavailable in real-world settings. Reinforcement learning (RL) has recently emerged as a promising alternative, requiring only the information if two images show the same scene or not. However, existing RL formulations such as RIPE rely on coarse binary rewards and carefully constructed negative training pairs, limiting training stability and descriptor discriminability. In this paper, we revisit RL-based keypoint learning and propose a reward that fully exploits the geometric consistency signal, deriving both reward and penalty from a single positive pair without contrasting against negatives. This richer signal provides sufficient supervisory contrast to learn discriminative detectors and descriptors from positive image pairs alone, enabling representation learning under extremely limited supervision. Furthermore, we show that the same RL objective can be extended to the matching stage by adapting LightGlue, raising AUC@5 on MegaDepth1500 from 56.58 to 59.65 and enabling weakly-supervised training of the full sparse matching pipeline from image pairs with partial visual overlap. We validate our approach on established benchmarks, demonstrating competitive results compared to fully-supervised methods. We further show that the method can be even trained on low texture medical video sequences, where camera poses are usually unavailable and standard SfM pipelines often fail. Code and data are available at https://github.com/fraunhoferhhi/RIPEpp .

Johannes Künzel, Peter Eisert, Anna Hilsmann

Fraunhofer Heinrich-Hertz-Institute, HHI, Germany · Humboldt University Berlin, Germany

Representation Learning Reinforcement Learning Image Matching Keypoint Detection Self-Supervised Visual Representation Learning

Aug 13, 2026cs.CV

A Controlled Study of Self-Supervised Image and Video Pretraining under Limited Resources

Visual foundation models are a cornerstone of image and video understanding but typically require large amounts of data and computation. The current scale required for pretraining visual foundation models may be unsustainable or unnecessary, and significant benefits arise when effective models can be obtained with fewer resources. To better understand how self-supervised learning (SSL) objectives behave under resource constraints, we conduct a controlled study of image and video SSL objectives under matched data, architecture, and compute budgets. We compare contrastive, reconstruction, feature-prediction, and diffusion objectives and evaluate both standalone and jointly trained image-video SSL formulations across a diverse set of image and video understanding tasks. Our results show that DINOv2-style pretraining consistently provides the strongest overall performance under limited resources. Furthermore, combining DINOv2 with video SSL objectives such as VideoMAE substantially improves image classification and segmentation performance, but degrades video tracking and camera-pose estimation performance, revealing an important tradeoff between semantic and geometric representation learning. These findings suggest that combining image and video SSL objectives can be beneficial in resource-limited settings, while highlighting the need for improved methods that better balance semantic, temporal, and geometric supervision.

Brunó B. Englert, Gijs Dubbelman

Eindhoven University of Technology

Vision Foundation Models Self-Supervised Pre-Training Video Representation Learning Self-Supervised Visual Representation Learning

Aug 11, 2026cs.CV

SAR2Agri: Learning SAR Intensity Representations for Agricultural Monitoring

Agricultural monitoring faces unique challenges, arising from the landscape's complex temporal, phenological, and climate dynamics, yet monitoring them is critical for ensuring food security. Synthetic Aperture Radar (SAR) satellites offer all-weather day-night imaging capability supporting key monitoring tasks including crop type mapping, yield prediction and phenological event detection. Existing multimodal remote sensing foundation models including TerraMind and CopernicusFM learn SAR representations by grounding them in optical imagery using joint encoding and contrastive learning techniques, while SAR-specific foundation models such as SAR-JEPA, SARMAE, and SAR-W-MixMAE primarily focus on target detection, flood mapping, and land cover classification applications. Recent work has introduced phenology inspired temporal pretext tasks with optical imagery which has shown strong performance on agricultural downstream tasks. In this work, we propose the first self-supervised learning pipeline focused on using only SAR intensity imagery for agricultural applications. We improve the temporal pretext tasks through masking and curriculum learning to enhance the pretraining pipeline's ability to capture phenological features from SAR. On the SICKLE benchmark, our final model achieves 84.9% IoU on crop type mapping, outperforming optical baselines (by 15.3 pt) and existing SAR baselines (by 2.2 pt), demonstrating the effectiveness of our proposed pipeline for pretraining SAR intensity encoders for agricultural monitoring.

Moti Rattan Gupta, Anupam Sobti

Plaksha University, Mohali, Punjab, India

Representation Learning Agricultural Remote Sensing Synthetic Aperture Radar Self-Supervised Visual Representation Learning

Aug 9, 2026cs.CV

Fourier Self-Supervision for Fine-Grained Generalized Category Discovery

Generalized Category Discovery aims to recognize known categories while identifying novel ones within unlabeled data. Existing methods, typically based on self-supervision and contrastive learning, often struggle to capture fine-grained distinctions, relying on superficial visual cues rather than the intrinsic attributes humans use for categorization. We introduce Fourier Self-Supervision, that leverages the Fourier transform of images to enhance the discrimination of subtle differences and support the discovery of new categories. Our method employs a dual frequency filtering strategy: a low-pass filter first extracts broad, abstract attributes that capture high-level category information, while a high-pass filter emphasizes fine details such as edges and textures that are essential for fine-grained recognition. Each operates on a dedicated latent space, and their overlapping representations together yield a richer, more complete feature space. This dual-frequency approach not only refines feature extraction to identify novel categories, but also strengthens the model's discriminative power in fine-grained category discovery. Experiments on multiple fine-grained datasets show that incorporating Fourier Self-Supervision outperforms state-of-the-art methods, even when the number of classes is unknown, demonstrating its effectiveness for Generalized Category Discovery. Our code is available at: https://github.com/SarahRastegar/FourEx.

Sarah Rastegar, Mina Ghadimi Atigh, Pascal Mettes +2

University of Amsterdam · University of Technology Nuremberg

Frequency-Domain Feature Learning Fine-Grained Image Classification Self-Supervised Visual Representation Learning Generalized Category Discovery

Aug 8, 2026cs.CV

Three Necessary Principles for Self-Supervised Visual Representation Learning

We argue that learning visual representations without labels requires a training signal jointly complete across three non-overlapping objectives: semantic invariance across augmented views, patch-level spatial prediction, and representational non-degeneracy. We formalize these as the observation, prediction, and regularization principles and prove (i) that combining observation and prediction without regularization admits the constant encoder as a global minimizer under negative-free alignment; (ii) that the two objectives are gradient-complementary and structurally non-conflicting at the encoder output; and (iii) that the momentum encoder converges to the same fixed point as the online encoder and provides no collapse guarantee at convergence. Contrastive alignment provides only self-limiting collapse resistance, formalized via an explicit gradient-decay argument. Dropping prediction withholds the spatial training signal by construction; dropping observation forfeits cross-view semantic invariance by construction; at the scale we study, no pair substitutes for the third. Every major self-supervised method is a special case of a single unified energy decomposition. We pair every theoretical claim with a controlled experiment, including a patch-retrieval evaluation for the spatial consequence of prediction.

Nikos Giakoumoglou, Paschalis Giakoumoglou, Tania Stathaki

Imperial College London · CERTH, ITI

Contrastive Learning Representation Collapse Self-Supervised Learning Self-Supervised Visual Representation Learning

Aug 7, 2026cs.CV

UniJEPA: A Unified Joint-Embedding Predictive Architecture for Task-Agnostic Visual World Modeling

Joint-Embedding Predictive Architectures (JEPAs) have emerged as a principled framework for self-supervised learning of world models in compact latent spaces, yet existing methods are fragmented: some predict masked parts of a single image in latent space (I-JEPA), others learn to predict global photometric transformations (Image World Models), while video-scale JEPAs predict future temporal states and are post-trained for action-conditioned planning (V-JEPA~2, DINO-World, DINO-WM). These objectives are treated as distinct recipes with separate encoders, predictors, and anti-collapse regularizers, hindering a single model from unifying image-level and video-level world modeling. We present UniJEPA, a unified JEPA that jointly learns photometric prediction (image-level transformations) and temporal prediction (video-level next-state dynamics) in one shared latent space. A single end-to-end objective, composed of a next-embedding prediction loss and a Gaussian regularizer, yields a provably anti-collapse encoder-predictor pair trainable from raw pixels without EMA, stop-gradient, or pre-trained encoders. We show that the same latent space supports controllable abstraction: photometric prediction learns invariant structure while temporal prediction learns equivariant dynamics. After action-conditioned post-training on offline trajectories, UniJEPA enables zero-shot planning by treating goal features as prediction targets. On image, video, and control benchmarks, UniJEPA matches or surpasses task-specific JEPAs while requiring a single loss hyperparameter, and plans up to tens of times faster than generative world models at comparable accuracy.

An Lanji, Dawei Liu, Jin Li +3

University of Electronic Science and Technology of China, Chengdu, China.

World Model Learning Joint-Embedding Predictive Architecture Model-Based Planning Predictive Representation Learning Self-Supervised Visual Representation Learning Video World Models

Aug 6, 2026cs.CV

Learning visual representations for compositional analysis of artworks and photographs

Composition, the deliberate arrangement of visual elements, is central to how meaning, emotion, and aesthetic quality are conveyed in artwork, yet it remains among the least formalized dimensions of visual understanding. Prior work highlights a persistent gap in learning meaningful compositional representations, attributing it to semantic bias and suggesting that human-inspired approaches may be key. We compare two parallel paradigms for composition analysis: a human-inspired method grounded in perceptual grouping, and fine-tuned foundation models enabled by recent large-scale compositional datasets. The human-inspired approach uses object-centric models for region-level decomposition and a graph attention network to capture spatial relationships between elements. Both paradigms are evaluated on composition score/category prediction, compositional image retrieval, and visual saliency detection. With frozen encoders, the human-inspired method achieves competitive performance while remaining interpretable. When sufficient data enables fine-tuning, large self-supervised models outperform significantly, but at the cost of interpretability and cross-domain generalization.

Fatemeh Behrad, Tinne Tuytelaars, Johan Wagemans

KU Leuven University Belgium

Visual Representation Learning Compositional Representation Learning Self-Supervised Visual Representation Learning Composed Image Retrieval

Aug 4, 2026cs.LG

Attention-Only White-Box Transformer via LeJEPA-Based Self-Supervised Pretraining

Existing studies on self-supervised learning for white-box networks typically decouple the derivation of white-box networks via optimization algorithms from self-supervised learning paradigms. In this work, we instead revisit the two components from a joint perspective. The LeJEPA-based self-supervised framework assumes an isotropic Gaussian distribution as the optimal embedding distribution for downstream tasks, which is conceptually equivalent to the expansion term

R(Z)

in the sparse rate reduction objective guiding white-box Transformer optimization. Building on this observation, we use the LeJEPA self-supervised paradigm to optimize

R(Z)

, and derive the remaining terms

R^{c}(Z\mid U_{[K]})+λ\lVert Z\rVert_{0}

via the alternating direction method of multipliers (ADMM) into an attention-only Transformer that dispenses with the ISTA structure or MLP layers of the original design. Experimental results demonstrate that our attention-only white-box Transformer achieves classification accuracies of

88.88\%

on CIFAR-10 and

63.54\%

on CIFAR-100 at the Base scale under the LeJEPA self-supervised paradigm, while the original white-box Transformer CRATE achieves classification accuracies of

89.18\%

on CIFAR-10 and

63.56\%

on CIFAR-100. Our model achieves competitive performance while reducing the parameter count by roughly

31\%

. Beyond the white-box setting, we further investigate standard ViTs and find that replacing all MLP blocks with ReLU activations under knowledge distillation removes approximately 66% of the parameters while preserving competitive accuracy, motivating further investigation into the potential redundancy of MLP modules in standard ViT architectures.

Yang Bai, Linyuan Wang, Haoyang Jiang +3

Information Engineering University, ZhengZhou,China

Vision Transformer Efficient ViTs Self-Supervised Visual Representation Learning

Aug 3, 2026cs.CV

Self-supervised DXA representations encode multi-system disease risk, biological aging and heritability

Whole-body dual-energy X-ray absorptiometry (DXA) scans are routinely acquired to measure bone density and regional body composition, leaving their spatial structure largely unused. Here, we show that self-supervised learning (SSL) can convert raw DXA images into representations of systemic health. We introduce LeDXA, a vision model based on a joint-embedding predictive architecture (JEPA) that learns by predicting latent representations rather than reconstructing pixels. Trained from scratch on 11,540 unlabeled Human Phenotype Project scans, LeDXA was evaluated internally and on 47,400 external UK Biobank (UKBB) scans. It improved cross-cohort prediction of prevalent diseases and biomarkers beyond scanner-derived DXA measurements and DINOv3, a state-of-the-art general-purpose model, despite approximately 150,000-fold fewer training images and nearly 40-fold fewer parameters. Over a median 4.3-year UKBB follow-up, LeDXA improved incident disease prediction over tabular DXA measures, with the largest gains for hip and knee arthrosis and type 2 diabetes. For hip arthrosis, 66% of incident cases occurred in the highest-risk quartile versus 41% for tabular measures. Its representations predicted chronological age externally (r = 0.88; mean absolute error = 2.90 years), and the biological-age gap tracked broader disease burden and a 45% higher mortality hazard in the oldest-appearing quartile. The gap also decreased in women after starting hormone-replacement therapy, suggesting it may be modifiable. Genome-wide associations recovered mostly known body-composition and bone-density loci, and LeDXA embeddings were more heritable than DINOv3's. These findings reveal prognostic information in DXA images that conventional readouts discard, learnable with relatively little data and modest compute.

Gil Sasson, Zachary Levine, Smadar Shilo +9

Department of Computer Science and Applied Mathematics, Weizmann Institute of Science, Rehovot, Israel. · Gray Faculty of Medical and Health Sciences, Tel Aviv University, Tel-Aviv, Israel. · The Jesse Z and Sara Lea Shafer Institute for Endocrinology and Diabetes, National Center for Childhood Diabetes, Schneider Children’s Medical Center of Israel, Petah Tikva, Israel. +5

Medical Imaging Medical Image Representation Learning Self-Supervised Visual Representation Learning Clinical Risk Prediction

Jul 30, 2026cs.CV

Physics-Aligned Self-Supervised Learning for Scientific Imaging

Data augmentations define the invariances learned by self-supervised learning (SSL). Standard augmentation pipelines were designed for natural images, yet scientific imaging modalities are governed by physical measurement processes with distinct symmetry and acquisition constraints. Enforcing invariances that contradict these constraints can distort learned representations and limit downstream performance, but practitioners moving from machine learning into a new scientific modality currently have little guidance beyond transferring natural-image pipelines unexamined. We address this gap with a principled, reproducible procedure for augmentation design in scientific SSL: we formalise the physics-aligned augmentation set as a union of measurement-consistent symmetries and acquisition-driven perturbations, and we give a concrete, largely label-free workflow---enumerate candidates, label each by the measurement operator, validate with representation-geometry diagnostics, and confirm by single-factor ablation---for selecting them. We instantiate the procedure for real-space electron microscopy and reciprocal-space 4D-STEM diffraction, and evaluate it across five SSL paradigms (DINOv2, SimCLR, MAE, VICRegL, I-JEPA) on classification and crystal-orientation regression. Physics-aligned augmentations substantially improve downstream performance for objectives relying on cross-view consistency, reduce geodesic error and improve robustness under realistic acquisition variability (detector gain, resolution loss), and systematically reshape representation geometry. While our experiments use electron microscopy, the procedure is modality-agnostic and applies to other measurement-driven domains such as medical and remote-sensing imaging. These results position augmentation design as a primary, and controllable, source of inductive bias in scientific self-supervised learning.

Bashir Kazimi, Stefan Sandfeld

Institute for Materials Data Science and Informatics (IAS-9), Forschungszentrum Jülich GmbH, 52425, Jülich, Germany · Chair of Materials Data Science and Materials Informatics, Faculty 5 – Georesources and Materials Engineering, RWTH Aachen University, 52056, Aachen, Germany

Data Augmentation Scientific ML Self-Supervised Visual Representation Learning Microscopy Image Analysis

Jul 28, 2026cs.CV

OrganLens: Organ-Specific Representation Learning for CT Foundation Models

A CT examination captures multiple organs, but many biomedical questions concern abnormalities, prognosis, or longitudinal change in a specific organ. These questions require a separate representation for each organ within the same CT volume. Existing CT foundation models commonly produce a single volume-level representation, while recent anatomy-aware methods either encode pre-separated organ volumes or explicitly disentangle images into organ token groups. The former may remove clinically relevant surrounding context, while the latter does not condition a shared encoder on a selected organ before its features are formed. We introduce OrganLens for organ-specific representation learning through self-supervision. An organ identity conditions a shared CT encoder, while organ-specific distillation and anatomy-mask supervision shape features for anatomy-weighted pooling into organ-specific representations. At inference, the shared model produces 11 organ-specific representations without external segmentation masks. We evaluate OrganLens on CT-RATE, RAD-ChestCT, INSPECT, and NLST across diverse acquisitions and downstream evaluations. Relative to CT-pretrained DINOv2, heart representations raise CT-RATE cardiomegaly AUROC from 0.910 to 0.953, while lung representations improve the Harrell C-index for NLST lung-cancer mortality by 14.2%. The global representation reaches INSPECT Recall@10 of 33.09% and 32.04% for text-to-image and image-to-text retrieval, respectively. Across organ-related tasks, anatomically matched representations provide stronger task-relevant signal, while the global representation retains broad utility. OrganLens offers a scalable approach to organ-specific CT representation learning with a shared encoder. More broadly, it provides the medical research community with a reusable framework for studying organ-specific disease across cohorts and clinical endpoints.

Zhixuan Ge, Anqi Li, Sadeer Al-Kindi +2

Rice University Houston, Texas, USA · Department of Cardiology DeBakey Heart and Vascular Center Houston, Texas, USA · University of Washington Seattle, Washington, USA

Medical Imaging Foundation Models Representation Learning Medical Image Representation Learning Computed Tomography Self-Supervised Visual Representation Learning

Jul 25, 2026cs.CE

A Scale-adaptive Vision Model Links C. elegans Neuronal Morphology to Behavior for Neurotoxicity Assessment

Neurological disorders are a leading cause of global disability and are increasingly linked to environmental chemical exposures. Yet neurotoxicity assessment still relies on hand-scored morphological readouts that are subjective and poorly predictive of behavioral outcomes. Caenorhabditis elegans provides a genetically tractable, 3R-compliant alternative, but quantifying neuronal phenotypes from confocal microscopy at scale remains computationally challenging: existing vision foundation models, trained on natural or radiological images, cannot resolve the sparse signals and multi-scale lesions of neuronal imaging. Here, we introduce a dedicated self-supervised vision model for C. elegans dopaminergic neurons, together with CeNeuMorph, a multi-grained confocal benchmark of 27,117 annotated images. Specifically, moving beyond standard Masked Autoencoders, we propose a scale-adaptive masked image modeling strategy that jointly learns representations across resolutions and patch sizes under a fixed token budget. By decoupling structural semantic learning from rigid grid constraints, the model effectively resolves the full spectrum of neurodegenerative lesions - ranging from fine dendritic beading to gross soma shrinkage - within a tractable computational framework. Finally, our model surpasses both generalist and biomedical foundation models across classification, segmentation and detection tasks. Fusing visual features with morphological descriptors enables prediction of dopamine-dependent behavioral deficits (

R^2=0.498

). Screening 180 agrochemicals, we identify the benzimidazole moiety as a previously unrecognized determinant of dopaminergic neurotoxicity. Together, the work demonstrates how scale-adaptive self-supervised learning can connect morphology to function for a scalable alternative to mammalian in vivo models for neurotoxicity assessment and drug discovery.

Haochao Ying, Shenchong Lv, Yutao Sun +8

School of Public Health and Second Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, 310058, China. · State Key Laboratory of Transvascular Implantation Devices and TIDRI, Second Affiliated Hospital of Zhejiang University School of Medicine, Hangzhou, 310009, China. · Polytechnic Institute, Zhejiang University, Hangzhou, 310058, China. +5

Vision Foundation Models Toxicity Detection Self-Supervised Visual Representation Learning Masked Image Modeling

Jul 23, 2026cs.CV

Self-Supervised Learning of Structured Dynamics from Videos

Understanding motion in video is a fundamental challenge for visual learning, as frame-to-frame change entangles two sources of dynamics: camera motion and object motion. This decomposition has remained underexplored in representation learning, partly because these factors are tightly coupled in natural videos and difficult to supervise separately. Yet recovering it is important for learning robust motion representations that separate meaningful object dynamics from camera-induced variation. We study whether such structured motion representations can be recovered from frozen features of a pretrained image vision transformer. We propose the Structured Dynamics Model (SDM), which explicitly separates the dominant source of temporal change from residual dynamics through future-feature prediction, rather than representing video change with a single entangled latent or with unstructured, spatially dense transition tokens. Training combines self-supervised learning on real video with weak supervision of scene dynamics on synthetic Kubric data. We evaluate SDM on ProbeMotion, a new evaluation suite spanning synthetic and real videos with camera motion, object motion, and combined dynamics. SDM outperforms backbone baselines using global CLS or average-pooled features, and compares favorably to strongly supervised representations such as VGGT on several probes, despite using substantially weaker supervision. These results suggest that pretrained image models can be readily repurposed into structured video-dynamics representations, providing a useful inductive bias for learning and analyzing latent video dynamics.

Lukas Knobel, Andrew Zisserman, Yuki M. Asano

Fundamental AI Lab, UTN · VGG, University of Oxford

Video Representation Learning Self-Supervised Visual Representation Learning

Jul 21, 2026cs.CV

scMIR: a vision-language foundation model for single-cell light microscopy image representation

Single-cell light microscopy images have become an important data source for characterizing cell phenotypes, but their complexity and heterogeneity pose challenges to high-throughput automated analysis. Existing representation learning methods mostly rely on task-oriented modeling, which is limited by specific datasets and predefined tasks, making them difficult to generalize across different cell types and microscopy modalities, and experimental conditions. Although general-purpose methods have improved the generalization ability of image representation in recent years, their limited utilization of experimental background and biological context information still poses challenges in complex phenotypic analysis. Here, we propose scMIR, a vision-language foundation model for single-cell light microscopy image representation. By synergistically combining self-supervised image reconstruction with text-guided cross-modal alignment, scMIR can simultaneously encode morphological and biological semantic information in a unified representation space. scMIR is pre-trained on 207,957 image-text pairs, covering various cell types, microscopy modalities, and perturbation conditions. scMIR outperforms existing general models and task-oriented methods as systematically evaluated on various complex tasks using 16 benchmark datasets, including cell classification, clustering, phenotype inference, and batch effect correction tasks. Furthermore, scMIR shows a strong generalization ability across various tasks without requiring task-specific fine-tuning. With its unique advantages, we envision scMIR may promote the standardization and automation of high-throughput phenotyping workflows through supporting various downstream analysis tasks.

Yifan Shang, Jiahui Tan, Xiangxiang Zeng +1

Laser Metrology and Biomedicine Lab, Department of Biomedical Engineering, The Chinese University of Hong Kong, Hong Kong, China · State Key Laboratory of Chemo and Biosensing, College of Computer Science and Electronic Engineering, Hunan University, Changsha 410082, China

Cross-Modal Representation Learning Visual Representation Learning Multimodal Pretraining Self-Supervised Visual Representation Learning Microscopy Image Analysis

Jul 15, 2026cs.CV

Emergent Region-Level Facial Correspondence in Frozen Vision Foundation Models

Frozen self-supervised vision models can align parts of generic objects, but it remains unclear whether this correspondence extends to human faces, where global layout is shared while identity-specific appearance varies sharply. We test whether frozen DINOv3 features define a region-level facial coordinate system: a feature space in which eyes, brows, nose, mouth, skin, and hair remain distinguishable across people and across time without face-specific training. Using DINOv3 ViT-L/16 patch embeddings and FaRL only as a face-part labeling interface, we evaluate cross-identity nearest-neighbor matching and temporal label propagation on 200 CelebDF-v2 real videos. DINOv3 achieves 83.0% region-level semantic accuracy under unconstrained cross-identity matching, compared with a 23.0% area-weighted random baseline, and 95.5% temporal tracking accuracy without a learned temporal module. A no-FaRL control collapses to 0.9%, showing that FaRL supplies semantic initialization while DINOv3 supplies dense spatial correspondence. The strongest correspondence appears at an intermediate layer: block 18 gives a 4.93x same-region versus cross-region discrimination ratio, compared with 1.48x at the final block. Against CLIP ViT-L/14, DINOv3 shows only a small aggregate advantage but a +16.8 pp gain on anatomical regions, indicating that image-level contrastive supervision captures coarse facial layout but not fine-grained anatomical identity. These results establish frozen DINOv3 as a strong zero-shot representation for region-level facial correspondence and identify intermediate self-supervised features as the most useful layer for dense face analysis.

Izaldein Al-Zyoud, Abdulmotaleb El Saddik

MCRLab, School of Electrical Engineering and Computer Science University of Ottawa, Ottawa, ON, Canada

Semantic Correspondence Vision Foundation Models Non-Rigid Shape Correspondence Self-Supervised Visual Representation Learning

Jul 14, 2026cs.CV

Self-Supervised Visual Representation Learning: Pretrain-Finetuning or Joint Training?

Self-supervision is a powerful technique for learning visual representations from unlabeled data. Existing techniques primarily adopt a two-stage approach for self-supervised learning (SSL): a pretraining stage on unlabeled data followed by a finetuning stage on labeled data. While this pipeline has demonstrated extreme effectiveness, the interaction between self-supervised and supervised learning objectives remains insufficiently understood. In this work, we systematically investigate whether jointly optimizing the self-supervised and supervised objectives during training provides a better alternative. We compare two training paradigms: (1) the aforementioned pretraining followed by finetuning (PFT) and (2) joint training (JT), where self-supervised and supervised losses are optimized simultaneously in the same network. Across eight representative SSL methods and diverse computer vision tasks on natural, medical, crisis response, and remote sensing data, we evaluate performance under varying percentages of labeled data. Our results reveal that the relative effectiveness of PFT and JT depends strongly on the task at hand, the availability of labeled data, and the complexity of the domain. We find that JT consistently improves data and training efficiency while being robust in low-label settings, while PFT is more reliable in more specialized domains. We further analyze representation quality, robustness, and cross-domain generalization, providing new insights into how self-supervised and supervised objectives interact during optimization. We establish a comprehensive empirical benchmark for hybrid SSL-based semi-supervised learning and offer practical guidance for selecting appropriate training strategies across diverse vision applications.

Nusrat Munia, Tyler Ward, Nishat Nayla +2

Department of Computer Science University of Kentucky Lexington, KY 40506, USA · Kentucky Geological Survey University of Kentucky Lexington, KY 40506, USA

Supervised Fine-Tuning Semi-Supervised Learning Self-Supervised Visual Representation Learning

Jul 14, 2026cs.CV

A Masked Autoencoder Approach to Unsupervised Steel Surface Defect Recognition

Automated visual inspection of steel surface defects is a recurring quality control task in which labeled defect data is scarce and costly to obtain, while unlabeled surface images are abundant, which motivates self supervised methods that learn useful representations without class labels. A Transformer based Masked Autoencoder is used here to learn representations of steel surface defects for unsupervised grouping. During pretraining, 75% of the input image patches are randomly masked, and a lightweight decoder reconstructs the masked regions from the visible 25%. The encoder is trained jointly with an auxiliary defect localization objective, used only as a training signal and not evaluated as a detector. The decoder reaches a structural similarity score of 0.92 and a mean squared error of 0.47. Features from the pretrained encoder are then clustered using UMAP for dimensionality reduction and Agglomerative clustering, reaching a Hungarian matched accuracy of 91.3% against the six known defect categories.

Shrey Patel

University of Maryland, College Park · United States of America

Masked Autoencoders Industrial Anomaly Detection Industrial Inspection Self-Supervised Visual Representation Learning

Jul 10, 2026eess.IV

Joint-Embedding Predictive Architecture for Solar PV Panel Fault Classification

The rapid expansion of solar photovoltaic (PV) systems has increased the need for reliable and scalable fault classification, as manual inspection is impractical at scale. Thermal infrared (IR) imaging provides a non-contact solution for identifying PV faults; however, accurate classification remains challenging due to class imbalance, limited texture information, and subtle thermal differences. In this work, we investigate the applicability of Joint-Embedding Predictive Architecture (JEPA) for thermal IR PV fault classification across various scenarios and propose JEFFNet (JEPA-EFFicientNet), a multibranch architecture that combines JEPA-based self-supervised representation learning with EfficientNetV2-S-based supervised convolutional feature extraction. JEFFNet fuses semantic representations from a JEPA-pretrained Vision Transformer with convolutional features from EfficientNetV2-S, enabling complementary feature learning. JEFFNet is evaluated on two public thermal IR datasets, PVF-10 and InfraredSolarModules (ISM), for both multiclass and derived binary (healthy/faulty) classification. On PVF-10, JEFFNet achieves an F1-score of

93.21

and an accuracy of

94.33

in the 10-class task, and an F1-score of

97.53

and an accuracy of

96.41

in the derived 2-class task. On ISM, JEFFNet achieves an F1-score of

72.60

and an accuracy of

83.88

in the 12-class task, and an F1-score of

94.69

and an accuracy of

94.78

in the derived 2-class task. JEFFNet also uses only 108.6M parameters versus 205.91M for GEPFNet, a 47.2% reduction. These results demonstrate that combining self-supervised semantic and supervised convolutional features provides an effective, parameter-efficient solution for thermal IR PV fault classification. The source code is publicly available at https://github.com/Azimi2kht/JEFFNet

Seyyedhamid Azimidokht, Mehdi Monemi, Abdelhak Kharbouch +4

University of Oulu, Oulu, Finland · Central Queensland University, Australia

Thermal Imaging Joint-Embedding Predictive Architecture Self-Supervised Visual Representation Learning Fault Diagnosis

Jul 8, 2026cs.CV

Lifelong Representations: A Survey on Continual Self-Supervised Learning for Vision Models

Traditionally, continual learning has assumed access to labeled data, yet many real-world applications -- such as lifelong robotics -- require models to adapt continuously from unlabeled streams. This has led to the development of continual self-supervised learning (CSSL), a rapidly growing area that lacks a dedicated, systematic review. In this work, we present a comprehensive survey of CSSL for vision, with connections to emerging vision-language settings. First, we analyze existing evaluation protocols and highlight inconsistencies that hinder fair comparison. We then examine why self-supervised objectives exhibit improved robustness to catastrophic forgetting, relating this to task-agnostic representations and smoother loss landscapes. Next, we organize existing methods into a unified taxonomy based on their forgetting-mitigation strategies, including distillation, replay, regularization, architectural approaches, model merging, and objective-level adaptation. Finally, we identify open challenges such as scalability and the need for fast adaptability. We argue that advancing CSSL requires moving beyond small-scale benchmarks towards continual pre-training paradigms for large-scale systems.

Sergi Masip, Alicja Dobrzeniecka, Jonathan Swinnen +4

PSI, KU Leuven, Leuven, Belgium · NASK - National Research Institute, Warsaw, Poland · Computer Vision Center, Universitat Autonoma de Barcelona, Barcelona, Spain +1

Self-Supervised Learning Self-Supervised Visual Representation Learning Catastrophic Forgetting

Jul 8, 2026cs.CV

`Attention-Guided Cross-Temporal Clustering for Self-Supervised Video Object Segmentation

Video object segmentation (VOS) is a fundamental task in video understanding, requiring accurate delineation and consistent tracking of objects across frames. While supervised methods achieve strong performance, they rely on densely annotated datasets that are costly to obtain and have limited domain coverage. Self-supervised learning offers a promising alternative by removing the need for manual labels; however, existing approaches often struggle to jointly maintain spatial accuracy and temporal coherence, particularly in unconstrained multi-object scenarios. Many rely on optical flow, synthetic motion cues, or task-specific pretraining, limiting scalability and generalisation. We propose a self-supervised framework, Cross-Temporal Consistency and Clustering, that learns mid-level, part-aware representations by combining attention-guided token selection with lightweight temporal clustering. Instead of operating at the pixel or whole-object level, the method aligns soft part assignments across time using a saliency-weighted symmetric consistency objective. The framework leverages a frozen transformer backbone with lightweight modules for adaptive token selection and multi-offset temporal alignment, enabling efficient scaling across resolutions and motion patterns.

Waqas Arshid, Mohammad Awrangjeb, Alan Wee-Chung Liew +1

School of Information and Communication Technology, Griffith University, Brisbane, QLD, Australia. · School of Engineering and Built Environment – Electrical and Electronic Engineering, Griffith University, Brisbane, QLD, Australia.

Video Object Segmentation Self-Supervised Learning Video Representation Learning Self-Supervised Visual Representation Learning

Jul 8, 2026cs.LG

Converge to Surprise: Evolutionary Self-supervised Image Clustering

A variety of self-supervised image clustering approaches are invented in the past years. However, all dominant approaches are exploitative: The direction of parameter updates is determined by known states (observed input samples and existing parameters). We propose an explorative self-supervised learning framework that steps out of this zone. We define a surprise score that measures how unlikely the model's output representation is, assuming that all pixels are i.i.d. random noise. Maximizing the surprise score forces the deep learning model to reject the random noise null hypothesis, or equivalently, to discover non-randomness from data. Also, we propose a fundamental assumption: a surprise score cannot, in general, be fully optimized by exploitative optimization approaches. Thus, we propose the converge-to-surprise scheme to optimize a model: an evolution-strategy (ES) outer loop, which maximizes the surprise score using the mutation-selection mechanism, paired with a periodic gradient-descent inner loop, which uses the surprising clusters already discovered by ES as surrogate targets. On simple image benchmarks, our framework trained from scratch achieves new state-of-the-art results in non-parametric self-supervised image clustering --- the strictest deep-clustering setting, where the number of classes is unknown during training.

Canlin Zhang, Xiuwen Liu

Independent Researcher · Department of Computer Science Florida State University

Unsupervised Learning Self-Supervised Learning Clustering Unsupervised Clustering Deep Clustering Image Clustering+1

Jul 7, 2026cs.CV

Breaking Spurious Correlations via Generative Randomization and Cross-Variant Self-Supervised Learning

Deep neural networks trained with Empirical Risk Minimization (ERM) often fail under distribution shifts because they exploit spurious correlations between object labels and background context. Recent generative approaches address this issue by creating counterfactual images with altered contexts, but typically use these samples as standard data augmentation, leaving the model free to retain background-sensitive representations. We propose a two-stage framework that uses generative intervention to explicitly learn background-invariant visual representations. First, we isolate the foreground object using zero-shot segmentation and generate context-shifted variants with a structure-preserving diffusion model, preserving object identity while varying the surrounding environment. We then introduce Cross-Variant Self-Supervised Learning, where variants of the same object under different backgrounds form positive pairs in a contrastive objective. This encourages the encoder to align object-centric representations while suppressing background-specific cues. Then, we fine-tune the pretrained encoder using an ERM warm-up followed by GroupDRO with layer-wise learning rates. Experiments on distribution-shift benchmarks demonstrate best worst-group performance, achieving 92.5% on Waterbirds, 81.7% on MetaShift, and 87.4% on NICO++. Code: https://github.com/surajyadav-research/GRSSL

Suraj Yadav, Anjaneya Sharma, Siddharth Yadav

Indraprastha Institute of Information Technology Delhi

Contrastive Learning OOD Generalization Spurious Correlation Robustness Counterfactual Data Augmentation Self-Supervised Visual Representation Learning

Jul 6, 2026cs.CV

Vision Pretraining for Dense Spatial Perception

Dense spatial perception is essential for physical intelligence, where visual systems are expected to recover structured, metric, and actionable representations from pixel observations. Modern visual foundation models tend to prioritize semantic invariance, often at the expense of detailed spatial understanding. In this work, we study vision pretraining through a boundary-centric lens, motivated by the premise that boundaries and shape discontinuities offer essential cues for perceiving geometric properties. Concretely, we propose masked boundary modeling, a self-supervised paradigm that dynamically learns sub-pixel boundary representations and subsequently leverages the discovered boundary-bearing tokens as masked targets to facilitate dense visual token learning. By scaling this framework, we develop LingBot-Vision and demonstrate its efficacy across a diverse set of downstream vision tasks with DINOv3 as a strong baseline. Remarkably, LingBot-Vision drives the progression from LingBot-Depth 1.0 to LingBot-Depth 2.0 for depth completion, and thereby yields enhanced depth estimation, a key pillar for embodied artificial intelligence. Our findings reveal that boundary modeling goes beyond simple line segments and instead serves as a scalable pretraining principle for learning spatially structured visual representations.

Zelin Fu, Bin Tan, Changjiang Sun +6

Depth Estimation Self-Supervised Visual Representation Learning

Jul 6, 2026cs.CV

Probing Geospatial SSL Representations with Environmental Signals

Self-supervised learning (SSL) is designed to learn generic, transferable representations rather than representations optimized for a single task. Most geospatial benchmarks evaluate representations solely through downstream tasks, providing limited insight into the information encoded within the representation itself. We ask a different question: do SSL representations of satellite imagery preserve statistical associations with environmental variables that co-vary with the imaging process? To answer this question, we probe SSL representations using co-located ERA5 reanalysis variables, a global dataset of physically consistent environmental variables, including temperature, precipitation, surface solar radiation, surface pressure, and volumetric soil water. These variables are physically related to the spectral reflectance and radar backscatter recorded by Sentinel-1 and Sentinel-2, making them meaningful evaluation targets despite not being used during SSL pretraining. We complement this probing analysis with intrinsic representation metrics to characterize representation geometry and investigate how these properties relate to downstream performance and the encoding of environmental signals. Using DINO, MAE, and MoCo models trained under identical conditions, we show that representation-level metrics distinguish models with similar downstream benchmark performance, providing complementary information beyond task-driven benchmarks. We further find that the linear accessibility of environmental signals is associated with performance on environmentally dependent tasks in the PANGAEA benchmark. Finally, we release ERA5 annotations co-located with the SSL4EO dataset to enable physically grounded representation evaluation for future geospatial foundation models.

Rohita Mocharla, Vishal M. Patel

Johns Hopkins Applied Physics Laboratory · Johns Hopkins University · Laurel, MD +1

Representation Geometry Geospatial Representation Learning Representation Probing Remote Sensing Benchmarks Self-Supervised Visual Representation Learning

Jul 5, 2026cs.CV

HASSL: Hierarchy-Aware Self-Supervised Learning Framework for Single Cell Microscopy

Hierarchical structure is common in image data, where fine-grained clusters often merge into larger, coarser semantic groups. In biological cell images, current self-supervised learning models often suppress this hierarchy, as coarse factors such as imaging modality can obscure finer morphological attributes in the latent space. We propose a hierarchy-aware self-supervised training framework to address this problem. Our method combines two components: a distillation framework with a segmentation teacher to improve morphological awareness in the latent space, and a hierarchy-aware contrastive loss based on HDBSCAN to improve decision boundaries between closely related subtypes at different hierarchical levels. Together, these components reduce the tendency of self-supervised learning to overemphasize coarse factors and instead align embeddings with semantic and morphological cues. This yields biologically meaningful sub-clusters driven by fine morphological detail. We train and evaluate our method on a curated corpus of 2.3 million single cells aggregated from 20 microscopy datasets, both labeled and unlabeled, covering 208 cell classes. Our method improves over baseline and counterpart methods, increasing average top-K accuracy by 2.8%, top-9 retrieval on the dataset with the deepest hierarchy by 6.3%, and downstream F1-score for biologically relevant drug classification from perturbed cell morphology by 7.8%.

Julius Riel, Vishwa Mohan Singh, Sai Anirudh Aryasomayajula +10

TUM.ai, Munich, Germany · Ludwig-Maximilian-University, Munich, Germany · Technical University of Munich, Munich, Germany +6

Contrastive Learning Hierarchical Representation Learning Medical Image Representation Learning Self-Supervised Visual Representation Learning Microscopy Image Analysis

Jul 5, 2026cs.LG

Mask-based Predictive Representations for Reinforcement Learning

Vision-based deep reinforcement learning involves dealing with high-dimensional inputs of image information. It is crucial to abstract effective states from high-dimensional image inputs and limited samples for sample-efficient reinforcement learning. To address this challenge, inspired by fields such as natural language processing and computer vision, we propose a self-supervised task based on mask prediction as an auxiliary task for reinforcement learning. This non-reconstruction method uses the sequence information collected by the agent from the environment and the context information in the sequence to predict the masked information, thereby strengthening the agent's understanding of the task and learning effective representations. Combined with transformers, we find that the model reconstructs the masked input sequence in the latent space. By feeding the compressed representations learned by this method into reinforcement learning models, we observe an improvement in the sample efficiency of reinforcement learning. Moreover, the model outperforms state-of-the-art sample-efficient reinforcement learning methods on multiple continuous and discrete control benchmarks.

Kai Zhao

School of Systems Science, Beijing Normal University · Beijing, China

Representation Learning Predictive Representation Learning Visual RL Sample-Efficient RL Self-Supervised Visual Representation Learning

Jul 4, 2026cs.CV

SiamJEPA: On the Role of Siamese Student Encoders in JEPA

Joint Embedding Predictive Architectures (JEPAs) have emerged as a promising framework for self-supervised representation learning by predicting latent embeddings of masked regions rather than reconstructing pixels. Existing JEPA methods typically employ a single student encoder, leaving the role of Siamese student encoders largely unexplored. In this paper, we propose Siamese JEPA (SiamJEPA), a JEPA framework with masked Siamese student encoders and an exponential moving average (EMA) teacher, which can also be viewed as a JEPA formulation of the brain-inspired representation learning model PhiNet. We further introduce Random Shuffle Teacher (RST), which removes spatial correspondence in teacher targets to encourage semantic patch representations, and develop an RST-based semantic-to-spatial curriculum. Experiments on ImageNet show that Siamese student encoders effectively regularize the JEPA objective, improving representation separability and accelerating early-stage learning. Moreover, under RST, stronger Siamese regularization substantially increases class-discriminative information in individual patch tokens, suggesting that the semantic bias arises from the interaction between RST and the Siamese objective rather than from shuffling alone. SiamJEPA consistently outperforms comparable single-encoder JEPA variants under limited training budgets. With RST-based curriculum learning and a ViT-Base backbone, SiamJEPA achieves 74.2% linear-probing accuracy after 450 epochs using a substantially simpler masking strategy, compared with I-JEPA (72.9%) and DSeq-JEPA (73.5%) after 600 epochs. These results demonstrate that Siamese student encoders provide an effective inductive bias for predictive representation learning and can be further enhanced through semantic-to-spatial curriculum learning. The source code is publicly available at https://github.com/oist/SiamJEPA.

Makoto Yamada

Okinawa Institute of Science and Technology

Joint-Embedding Predictive Architecture Siamese Neural Networks Predictive Representation Learning Self-Supervised Visual Representation Learning

Jul 2, 2026cs.CV

Mask-supervised Object-centric Representation Learning with LeJEPA

Self-supervised image encoders deliver strong features for downstream tasks but need many images for training. A natural remedy to counter this is to make each image count for more. A scene contains many objects, and given masks from human annotators or an off-the-shelf segmentation model, pre-training can focus on aligning per-object rather than image-wide representations, extracting more signal from every image. Existing mask-supervised methods do this through reconstruction or contrastive losses that leverage negative objects. We instead use two separate projection spaces for the alignment. In a \emph{semantic space}, per-object representations from different views are aligned. To avoid collapse, instead of using negative objects, which requires category definitions, we extend the negative-free LeJEPA objective and show that its distributional anti-collapse regularizer ports naturally from whole images to the variable-sized set of objects in a scene. In an \emph{instance space}, a contrastive loss separates per-object representations from their context and co-occurring instances, including those of the same category. To separate object representations from their context, we copy objects and paste them into other contexts, where each pasted copy serves as an additional view of the original object. Trained on COCO with ground-truth masks, our method outperforms image-level and mask-guided baselines on tracking (DAVIS), classification (ImageNet-1k) and re-identification (NAVI), matches the best of them on semantic segmentation (ADE20k) and keeps its lead over image-level LeJEPA and a supervision-matched alternative on COCO fractions down to 256 images.

Jakob Geusen, Ender Konukoglu

Biomedical Image Computing Group, ETH Zurich

Contrastive Learning Representation Learning Joint-Embedding Predictive Architecture Object-Centric Representation Learning Self-Supervised Visual Representation Learning