Self-Supervised Visual Representation Learning

Follow this topic Ask about this topic

Momentum

20 papers in the last four weeks, up 100% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 151

Jul 2, 2026cs.CV

Understanding Geometric Representations in Self-Supervised Vision Transformers via Subspace Intervention

We introduce a controlled subspace intervention framework to investigate how self-supervised Vision Transformers (ViTs) encode dense geometric information. While linear probing is widely used to assess geometric representations, it treats features as a black box, failing to disentangle the underlying topology. To address this issue, we decompose the weights of converged linear probes to isolate the low-rank subspaces containing explicit geometric signals using Singular Value Decomposition (SVD). Our perspective yields three key insights: (1) Pre-training objectives determine how features are encoded. DINOv2 aligns spatial features for efficient linear extraction, while Masked Autoencoders (MAE) tend to disperse these signals, requiring a broader spatial context. (2) Explicit geometric representations are highly compressible, suggesting dense predictive heads could potentially be constrained to low-rank subspaces with minimal performance loss. (3) The layer-wise task affinity suggests that geometric precision peaks at intermediate layers before yielding to semantic abstraction in the final layers. By connecting internal encoding mechanics with downstream performance, these findings provide a basis for effective feature selection and lightweight decoder design. The source code is available at https://github.com/Zhou-Weichen/Geosubprobe.

Weichen Zhou, Yawen Zou, Chunzhi Gu +3

University of Toyama, Toyama, Japan · University of Fukui, Fukui, Japan · Chukyo University, Aichi, Japan +1

Neural Representation Geometry Linear Probing Vision Transformer Geometric Representation Learning Self-Supervised Visual Representation Learning

Jul 1, 2026cs.CV

Mirror-Fusion Attention for Reflection-Aware Self-Supervised Representation Learning

Most self-supervised learning (SSL) methods encourage invariance across augmentations, but strict flip invariance can suppress informative left--right correspondences in approximately bilateral data such as medical images and human faces. We propose Mirror-Fusion-Augmented Self-Supervised Learning (MFASSL), a Vision Transformer framework that injects a soft reflection prior into standard SSL without redesigning the backbone. MFASSL constructs mirror-paired views aligned to an estimated symmetry axis and introduces a lightweight Mirror-Fusion Attention (MFA) module for adaptive token-level interaction between mirrored regions while preserving asymmetric cues. The base SSL objective is further coupled with reflection-consistency and mid-layer token-alignment losses. Across CheXpert, BraTS, CelebA-HQ, and WFLW, MFASSL improves downstream performance, calibration, and reflection robustness over MoCo-v3, DINO, and MAE baselines under matched ViT-B/16 settings. It also achieves stronger and more consistent gains than recent equivariant SSL approaches with only approximately 2.7% additional parameters. These results show that lightweight geometry-aware priors can effectively complement invariance-based SSL.

Ruixin Li, Jin Liu, Yuling Shi +1

Department of Computer Science and Engineering, University of Bologna, Bologna, Italy · Jilin University, Changchun, China · Shanghai Jiao Tong University, Shanghai, China

Representation Learning Vision Transformer Invariant Representation Learning Medical Image Representation Learning Self-Supervised Visual Representation Learning

Jul 1, 2026cs.LG

Task-Relevant Representation Decoupling for Visual Reinforcement Learning Generalization

Visual Reinforcement Learning (VRL) has achieved considerable success in solving control tasks. However, generalizing learned policies to new environments remains a major challenge, as agents often overfit to task-irrelevant features in the training environment. To solve this problem, we introduce the concept of decoupling observations into task-relevant and task-irrelevant representations. Building on this idea, we propose a self-supervised Task-Relevant Representation Decoupling (T2RD) algorithm for VRL. This algorithm consists of three components: task-relevant representation consistency, cross-reconstruction, and cross-dynamic prediction. The first two components achieve the decoupling of content and style features, but the resulting content representations are not necessarily task-relevant. To further refine task-relevant features from content representations, we design the third component that introduces dynamic prediction. T2RD achieves State-Of-The-Art (SOTA) generalization performance and sample efficiency in the DeepMind Control Suite and Robotic Manipulation tasks.

Jinwen Wang, Youfang Lin, Xiaobo Hu +4

Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence, School of Computer Science & Technology, Beijing Jiaotong University, China · CSSC Intelligent Innovation Research Institute, China and Zhejiang University, China

Representation Learning Representation Learning for RL Visual RL Generalization in RL Self-Supervised Visual Representation Learning

Jun 30, 2026cs.CV

MuSViT: A Foundation Vision Model for Sheet Music Representation

Foundation models have transformed vision and language processing by providing rich, reusable representations that transfer across diverse tasks. Sheet music, as a visual encoding of musical language, lacks such a strong domain-specific backbone. We introduce MuSViT (Music Score Vision Transformer): the first foundation vision model for sheet music representation -- a ViT encoder pre-trained via Masked Autoencoders on 9.7 million pages from the IMSLP. To handle the complexity of real-world scores, we adopt a two-stage curriculum: a synthetic warm-up on typeset scores followed by large-scale training on the full IMSLP corpus. We evaluate MuSViT on four downstream tasks -- full-page and staff-level music score recognition, music symbol detection, and score difficulty classification -- under two scenarios: linear probing (frozen encoder) and fine-tuning. Under linear probing, MuSViT consistently outperforms modern vision encoders, revealing that general-purpose representations, regardless of scale, fall systematically short on the structured symbolic properties of musical notation. Under fine-tuning, MuSViT generally improves upon task-specific state-of-the-art methods. An additional embedding-transcription consistency analysis reveals that MuSViT encodes symbolic musical structure directly in its representation space -- unlike other encoders, whose embeddings do not correlate with music notation content. These results establish MuSViT as a foundation backbone for sheet music understanding.

Carlos Penarrubia, Antonio Rios-Vila, Eliseo Fuentes-Martinez +4

Pattern Recognition and Artificial Intelligence Group, University of Alicante, Spain

Optical Music Recognition Representation Learning Vision Foundation Models Self-Supervised Visual Representation Learning

Jun 30, 2026cs.CV

Rhythm-Structured Predictive Learning for Remote Photoplethysmography

Remote photoplethysmography (rPPG) estimates physiological signals from facial videos by analyzing subtle pulse induced skin color variations. Despite recent progress, existing self-supervised rPPG methods mainly reconstruct masked pixels or low-level visual representations, which can bias the model toward facial appearance rather than latent physiological dy namics. Moreover, most recent Mamba-based approaches scan facial video tokens only in chronological order, limiting their ability to exploit the cyclic structure of pulse signals. To ad dress these limitations, we propose RhythmJEPA, a rhythm structured joint-embedding predictive learning framework for rPPG. Instead of reconstructing RGB frames, RhythmJEPA predicts latent teacher representations from masked facial videos, thereby encouraging physiology-aware representation learning in the embedding space. To explicitly model pulse-related tem poral structure, we introduce a Cyclic Rhythm-State Plan ner (CRSP), which estimates frame-wise latent physiological states and decodes the most plausible cyclic state path via dynamic programming with a constrained transition grammar. Guided by the decoded states, we further design a Dual Order Mamba Encoder (DOM), which combines conventional chronological scanning with state-ordered scanning to capture both local temporal continuity and long-range rhythm-consistent dependencies. Finally, a lightweight Spatial Pulse Mixer (SPM) extracts compact pulse-sensitive facial tokens with a favorable balance between complexity and performance. Experiments on PURE, UBFC-rPPG, and MMPD show competitive performance over representative rPPG methods. The codes are available at https://github.com/deconasser/RhythmJEPA.

Ba-Thinh Nguyen, Huu-Dung Nguyen, Thi-Duyen Ngo +1

VNU University of Engineering and Technology, Hanoi, Vietnam

Remote Photoplethysmography Mamba Joint-Embedding Predictive Architecture Predictive Representation Learning Photoplethysmography Self-Supervised Visual Representation Learning

Jun 30, 2026cs.CV

Mitigating Positional Leakage in 3D Masked Autoencoders for Robust Representation Learning

Masked autoencoding has emerged as a prominent paradigm for self-supervised learning on 3D point clouds, achieving competitive performance across downstream tasks. Unlike its 2D counterpart, 3D masked autoencoding directly reconstructs spatial coordinates, making it inherently susceptible to positional leakage. In this work, we identify that the decoder in existing 3D MAE frameworks tends to over-rely on positional information, which weakens semantic representation learning and leads to suboptimal feature quality. To address this issue, we propose MPL-MAE, a masked point learning framework that mitigates positional over-reliance while enhancing the utilization of encoder features. Specifically, we introduce a recalibrated positional embedding module that suppresses metric-dominant coordinate signals while preserving geometric topology, together with a gated positional interface module that dynamically regulates positional injection during reconstruction. These designs promote a more balanced interaction between spatial priors and semantic features, yielding robust and informative representations. Extensive experiments across downstream tasks demonstrate that MPL-MAE consistently achieves competitive performance, validating its effectiveness. Code is available at https://github.com/yanx57/MPL-MAE.

Xu Yan, Huiqun Wang, Chen Wang +2

SKLCCSE, Beihang University, Beijing, China · SCSE, Beihang University, Beijing, China · NERCBDS, EIRI, Tsinghua University, Beijing, China +1

Masked Autoencoders Point Cloud Learning 3D Representation Learning Self-Supervised Visual Representation Learning

Jun 30, 2026cs.CV

CLIMB: Centroid-Based Hierarchical Memory for Online Continual Self-Supervised Learning

Online Continual Self-Supervised Learning (OCSSL) aims to learn representations from a continuous stream of unlabeled data, without knowledge of task boundaries and under memory constraints. Existing methods rely either on replay buffers that exploit latent space structure, or on regularization alone. We present CLIMB (Continual Learning with Intelligent Memory Bank), which combines both simultaneously. Our method introduces a hierarchical centroid-based memory, bounded in total number of stored images, combined with knowledge distillation on replayed examples to limit representation drift. The memory groups similar images into centroids, providing hard-to-discriminate examples for contrastive learning while covering the diversity of observed distributions. Experiments on Split CIFAR-100 and Split ImageNet-100, on standard benchmarks from the state-of-the-art as well as a new protocol with irregular task distributions show that CLIMB outperforms state-of-the-art OCSSL methods.

Julien Lefebvre, Stefan Duffner, Mathieu Lefort

Universite Claude Bernard Lyon 1, CNRS, INSA Lyon, LIRIS, UMR 5205, 69622 Villeurbanne, France · INSA Lyon, CNRS, Universite Claude Bernard Lyon 1, LIRIS, UMR 5205, 69621 Villeurbanne, France · Universite Claude Bernard Lyon 1 / Univ. Rennes, Inria, CNRS, LIRIS UMR 5205 / IRISA UMR 6074, France

Self-Supervised Learning Replay-Based Continual Learning Online Continual Learning Incremental Learning Self-Supervised Visual Representation Learning

Jun 30, 2026cs.CV

ExPLoRe: Expert Patch-Level Loss Routing for Multi-Objective Masked Image Modeling

Multi-objective masked image modeling (MIM) combines complementary learning signals (token distillation, CLS alignment, and pixel reconstruction) but existing methods weight these objectives with global scalars, ignoring spatial heterogeneity across patches. We present ExPLoRe (Expert Patch-Level Loss Routing), which repurposes Soft Mixture of Experts (MoE) dispatch weights as learned, per-patch loss coefficients. The key mechanism is loss-coupling: allowing loss gradients to flow through dispatch weights to the router enables content-dependent specialization, where different patches receive different emphases across objectives. A detach ablation confirms loss-coupling as the core mechanism, degrading performance by 1.6% when gradients are blocked. On ImageNet-1K with ViT-Base, ExPLoRe improves over non-MoE baselines on two objective combinations (Token+CLS: +0.5% k-NN, +4.4% linear probe; Token+Pixel: +2.2% k-NN), achieving 80.6% linear probe and 85.3% finetuning accuracy, competitive with published methods. For downstream transfer, we develop adaptation recipes (Freeze Routing, Expert Dropout, and Freeze Attention) that improve MoE finetuning by +1.5% over the vanilla MoE, and close a 2.5--2.9 mIoU segmentation gap so that MoE models match or exceed non-MoE baselines on ADE20K.

Konstantinos Georgiou, Maofeng Tang, Hairong Qi

Min H. Kao Department of Electrical Engineering and Computer Science, The University of Tennessee, Knoxville, TN 37996, USA

Adaptive Loss Weighting Mixture-of-Experts Models Self-Supervised Visual Representation Learning Masked Image Modeling

Jun 28, 2026cs.CV

From Phase to Phenomenon: Self-Supervised Learning of Subsurface Scattering with Minimal Phase-shift Inputs

We propose a self-supervised pretraining framework for learning sub-surface scattering (SSS) light transport representations from minimal input. Our method leverages a stereo projector-camera setup that captures only eight high-frequency phase-shift profilometry (PSP) images per view to pretrain an encoder in a multi-view, multi-object setting. We introduce a tailored augmentation strategy for PSP-based SSS data, and show that it significantly outperforms standard ImageNet-style augmentations for SSL pretraining. The pretrained encoder learns generalizable SSS representations that transfer effectively to downstream tasks, including spatially varying relighting and representation evaluation using a kNN classifier. Combined with a decoder, the model reconstructs dense scattering footprint responses, trained using a dedicated cost function that improves accuracy, particularly for anisotropic footprints. Despite using only eight input images per view, our approach generalizes to unseen objects with complex geometry and material properties, achieving high-fidelity reconstructions while requiring orders of magnitude fewer images than prior methods.

Arjun Majumdar, Raphael Braun, Andreas Engelhardt +1

Eberhard Karls Universität Tübingen, Germany

Inverse Rendering Image Reconstruction Image Relighting Self-Supervised Visual Representation Learning

Jun 28, 2026cs.CV

The Platonic Defense: Backdoor Defense for Self-Supervised Encoders in the Era of Large Scale Pre-training

Self-supervised learning (SSL) pretrained models have become a dominant paradigm for visual representation learning, but they are vulnerable to backdoor attacks. Existing defenses struggle to defend against such attacks in a fully black-box setting because they often require access to labels, attack patterns, or training data. To tackle this issue, we propose a new attack-agnostic, model-agnostic, and modality-agnostic black-box test-time defense paradigm, called \emph{Platonic Representation Defense}. It is inspired by the Platonic Representation Hypothesis, which suggests that large-scale independently trained encoders converge toward compatible projections of the same underlying reality. We formalize this idea as a conditional energy function defined over source representations and a set of reference representations. The energy function is trained for detection through noise-contrastive estimation and for representation purification through denoising score matching. Theoretically, the energy gap between matched and mismatched samples is lower bounded by the mutual information between source and reference representations. We demonstrate the effectiveness of our method on multiple self-supervised encoders and more than 10 attacks. The method can perform both representation detection and purification, and achieves substantial performance gains across multiple attacks. Code is available here.

Tuo Chen, Minjing Dong, Benlei Cui +2

Southeast University · Ant Group · City University of Hong Kong +2

Adversarial Attacks Self-Supervised Visual Representation Learning

Jun 28, 2026cs.CV

How many labels do you need? A decision framework for cross-habitat marine species recognition

Automated image recognition is increasingly used to scale ecological monitoring beyond manual annotation, yet ecologists lack evidence-based guidance on how much labelling effort reliable deployment at new sites requires. We present a decision framework quantifying the trade-off between labelling effort and recognition accuracy when transferring vision systems across marine habitats. The benchmark spans five datasets, three oceans, and three taxonomic groups (fish, corals, invertebrates), from tropical reefs in the Great Barrier Reef and French Polynesia to a temperate Danish fjord. We evaluated four recognition models (DINOv2, CLIP, ResNet-50, EfficientNet-B4) under four adaptation strategies (linear probing, LoRA, Visual Prompt Tuning, full fine-tuning) across three protocols: within-habitat transfer across 20 reef sites (240 runs), cross-dataset geographic transfer along a difficulty gradient (40 runs), and few-shot adaptation curves with 0-100 labelled samples per class (648 runs). Frozen self-supervised foundation features (DINOv2 + linear classifier, 1,538 trainable parameters) generalised to unseen reef sites at least as well as fully fine-tuned convolutional baselines four orders of magnitude larger; they learned species-diagnostic, habitat-invariant representations, whereas baselines encoded habitat-specific shortcuts that fail at new sites. As few as 10-20 labelled images per species sufficed to deploy reliable recognition at a new site, cutting annotation effort by roughly an order of magnitude. Solution. Programmes expanding to new sites can deploy reliable recognition by pairing a frozen, open foundation model (DINOv2) with a simple linear classifier and annotating only 10-20 images per species - roughly 1-4 hours per site. The framework lets programmes budget labelling effort against expected accuracy across sites, ecosystems, and platforms.

Alzayat Saleh, Mostafa Rahimi Azghadi

College of Science and Engineering, James Cook University, Townsville, QLD, Australia · Centre for AI and Data Science Innovation, James Cook University, Townsville, QLD, Australia

Domain Generalization Vision Foundation Model Adaptation Few-Shot Image Classification Self-Supervised Visual Representation Learning Image Classification

Jun 28, 2026eess.IV

A Self-Supervised Learning Framework for Video Encoding Complexity Clustering

Adaptive video streaming is a widely used technique for delivering video content over the internet. One of the key challenges is determining the optimal encoding settings for each video, which can vary significantly based on its content and characteristics. In this paper, we propose Compression Echo Contrastive Learning (CECL), a novel self-supervised learning framework for clustering videos based on their encoding complexity. Our method leverages the response of a video to compression - the Compression Echo - as a supervisory signal, allowing the model to capture underlying encoding characteristics during pretraining. We conduct extensive experiments to demonstrate the effectiveness of our learned representations for the downstream task of clustering videos by their encoding complexity. Our results show that CECL improves upon existing state-of-the-art visual encoders and delivers strong bitrate and quality savings against the fixed bitrate ladder.

Krishna Srikar Durbha, Hassene Tmar, Ping-Hao Wu +2

The University of Texas at Austin · Meta Platforms, Inc. · University of Colorado at Boulder

Video Compression Self-Supervised Visual Representation Learning

Jun 28, 2026cs.CV

Spatially Localized Image Degradation Embeddings for Image Quality Assessment

Self-supervised learning (SSL) currently drives state-of-the-art performance in no-reference image quality assessment (NR-IQA). However, standard SSL pipelines uniformly apply synthetic distortions across the entire image field, which can limit their sensitivity to spatially localized and co-occurring degradations encountered in real-world content. In this work, we empirically expose this representational blind spot across existing state-of-the-art encoders, demonstrating their reduced sensitivity to spatially bounded image degradations. To bridge this gap, we introduce Spatial Localized Image Degradation Embeddings for Image Quality Assessment (SLIDE-IQA). SLIDE-IQA employs a dual-branch Vision Transformer framework that injects spatially bounded degradations into a contrastive pretraining objective. To handle the spatial complexity of these degradations, we introduce a Threshold-Bounded Exclusion Mechanism, a representational design choice that resolves structural conflicts arising from spatially localized distortions to ensure the latent space respects both degradation type and spatial scale. Finally, we show that SLIDE-IQA's synthetic-only pretraining significantly improves sensitivity to localized distortions, while achieving competitive performance on NR-IQA benchmarks against existing SSL NR-IQA models.

Krishna Srikar Durbha, Hassene Tmar, Ping-Hao Wu +2

The University of Texas at Austin · Meta Platforms, Inc. · University of Colorado Boulder

Image Quality Assessment Self-Supervised Visual Representation Learning

Jun 24, 2026cs.RO

TacVerse: A Multi-Sensor Dataset and Benchmark for Cross-Sensor Vision-Based Tactile Perception

Vision-based tactile sensors (VBTSs) enable robots to infer contact geometry and force-related cues by imaging deformation through an internal camera, yet generalisation across sensor designs remains poorly understood. We present TacVerse, a multi-sensor dataset and benchmark for cross-sensor vision-based tactile perception. The dataset contains 106,800 tactile images from seven VBTSs and supports three downstream tasks: shape classification, grating classification, and force regression. Experiments are conducted under three settings: within-sensor training, zero-shot cross-sensor transfer, and few-shot adaptation. Strong within-sensor performance across all tasks indicates that the collected tactile observations are informative for the target objectives. Direct cross-sensor transfer, however, leads to substantial degradation. Shape classification is comparatively robust, whereas grating classification and force regression are more sensitive to sensor shift. Few-shot adaptation for force regression consistently improves performance on unseen target sensors but does not fully close the gap to within-sensor upper bounds. A representation study further shows that MAE (Masked Autoencoder) pretraining provides the most consistent gains across tasks and sensors. TacVerse provides a controlled testbed for studying sensor shift, data-efficient adaptation, and self-supervised learning in tactile perception.

Lan Wei, Gurmeher Khurana, Sirine Bhouri +6

Imperial College London, UK. · Queen Mary University of London, UK · King’s College London, UK

Vision-Based Tactile Sensing Few-Shot Learning Robotic Tactile Sensing Self-Supervised Visual Representation Learning

Jun 23, 2026cs.CV

Tri-Efficient Transfer Learning for Point Cloud Videos

While point cloud foundation models have significantly advanced point cloud video understanding, existing parameter-efficient fine-tuning (PEFT) methods still suffer from two critical limitations: prohibitive annotation costs for large-scale point cloud datasets and severe memory bottlenecks. In this paper, we aim to mine richer supervision signals from existing data rather than blindly scaling datasets. A further key principle is that the memory footprint of fine-tuning must be drastically reduced compared to full fine-tuning, which remains elusive for current PEFT techniques. Driven by these challenges, we identify three core desiderata: data-, parameter-, and memory efficiency, and present PoinTriE, a unified framework that excels along all three dimensions. For pre-training, pseudo-motion trajectories are synthesized via rigid transformations, paired with text corpora and 2D projections derived from raw point clouds. We then propose a Geometric-Motion Duality Network optimized via multimodal contrastive learning, rigid rotation prediction, and motion distribution divergence to produce dense self-supervision. During fine-tuning, we freeze the pretrained backbone and only update a lightweight Spatio-temporal Side Network built with LoRA units. Equipped with a gradient flow masking strategy, PoinTriE simultaneously reduces memory consumption and parameter overhead. Extensive experiments confirm that PoinTriE establishes new state-of-the-art results on action recognition and semantic segmentation tasks.

Yiding Sun, Dongxu Zhang, Jihua Zhu +6

School of Software Engineering, Xi’an Jiaotong University · School of Information and Communication Engineering, Xi’an Jiaotong University · SIGS, Tsinghua University

Point Cloud Understanding Self-Supervised Visual Representation Learning Parameter-Efficient Fine-Tuning Data-Efficient Learning

Jun 22, 2026cs.LG

3D Masked Autoencoders are Robust Learners of Volumetric and Multimodal Cellular Representations for Microscopy

Self-supervised learning in fluorescence microscopy often relies on 2D projections, despite the inherently three-dimensional nature of cells. We present a systematic comparison of 2D and 3D masked autoencoders (MAE-2D vs. MAE-3D) on volumetric microscopy data. Under matched architectures and training protocols, MAE-3D consistently outperforms 2D max-projection and slice-based variants on downstream single-cell tasks. We further align visual representations with a pretrained protein language model (ESM2) and show that cross-modal supervision yields larger gains for volumetric models. Channel cross-attention and frequency-domain regularization are critical for leveraging 3D spatial context. On protein--protein interaction prediction, our best model achieves a ROC--AUC of 0.86, while on protein localization it reaches an AUC

_{\text{micro}}

of 0.95 and an F1

_{\text{micro}}

of 0.74, demonstrating competitive performance on both tasks. Overall, our findings highlight the potential of volumetric modeling and multimodal alignment for representation learning in single-cell microscopy.

Amirhossein Kardoost, Lion Gleiter, Tingying Peng +1

Institute of AI for Health & Helmholtz AI, Computational Health Center, Helmholtz Munich – German Research Center for Environmental Health, Neuherberg, Germany · Munich Center for Machine Learning (MCML), Munich, Germany · Department of Medicine III, Ludwig-Maximilian-University Hospital, Munich, Germany +2

Cross-Modal Representation Learning Masked Autoencoders 3D Representation Learning Self-Supervised Visual Representation Learning Microscopy Image Analysis

Jun 22, 2026cs.CV

UniverSat: Resolution- and Modality-Agnostic Transformers for Earth Observation

Vision Transformers (ViT) dominate computer vision. However, their reliance on rigid patch projectors hinders transfer to Earth Observation (EO), where input modalities, scales, and resolutions vary widely. We introduce UniverSat, a ViT-style backbone built around a Universal Patch Encoder that maps patches from arbitrary spatial, spectral, and temporal resolutions, and from both optical and non-optical sensors, into a shared embedding space with a shared set of weights. This enables training a single model on heterogeneous multimodal corpora via self-supervision, yielding robust, sensor-agnostic spatial features. We validate this approach with strong results across classification and segmentation on standard EO benchmarks from GeoBench, PANGEABench, and SpectralEarth. Our code and models are available at https://github.com/gastruc/UniverSat.

Yohann Perron, Guillaume Astruc, Nicolas Gonthier +2

LIGM, Ecole Nationale des Ponts et Chaussées, IP Paris, Univ Gustave Eiffel, CNRS · EFEO · LASTIG, Univ Gustave Eiffel, IGN, ENSG +2

Geospatial Representation Learning Vision Transformer Remote Sensing Self-Supervised Visual Representation Learning

Jun 22, 2026cs.CV

Evaluating self-supervised echocardiographic representations across downstream extraction strategies for left-ventricular segmentation and ejection fraction estimation

Self-supervised learning (SSL) is increasingly used in medical imaging to reduce annotation requirements, but representation quality is often judged using a single downstream evaluation setting. For dense clinical tasks, this can confound representation quality with the capacity of the downstream model used to recover task-relevant information. We present a systematic evaluation of self-supervised representations for left-ventricular segmentation and ejection fraction (EF) estimation from apical four-chamber echocardiography on EchoNet-Dynamic. Rather than relying on a single downstream probe, we compare a hierarchy of extraction strategies with increasing expressivity: heuristic extraction without mask-supervised training, frozen linear probes, frozen lightweight decoder probes, and partial fine-tuning. We apply this framework to two complementary representation families: generic frozen self-DIstillation with NO labels (DINOv3) features and a task-adapted dense self-supervised representation, Bootstrap Your Own Segmentation (BYOS). In both families, heuristic extraction substantially understated what was recoverable from the frozen representation. For DINOv3, performance improved from Dice 0.684 and EF mean absolute error (MAE) 13.01 under heuristic extraction to Dice 0.906 and EF MAE 9.65 with a frozen lightweight decoder, approaching a supervised U-Net baseline (Dice 0.915, EF MAE 9.72). For BYOS, performance improved from Dice 0.687 and EF MAE 17.83 under heuristic extraction to Dice 0.902 and EF MAE 8.74 with a frozen lightweight decoder. These results show that conclusions about self-supervised representation quality in dense echocardiographic analysis depend strongly on the downstream extraction strategy used for evaluation. We therefore argue that multi-strategy evaluation is an important methodological consideration for SSL in dense medical image analysis.

Sylwia Majchrowska, Philip Teare

aR&D Data ScienceSylwiaSkills &Majchrowskaa,Partnership, DataPhilipScienceTeareb& AI, BioPharma R&D, AstraZeneca, Pepparedsleden 1, Mölndal, 431 83, Sweden · bCentre for AI, Data Science & AI, BioPharma R&D, AstraZeneca, BiomedicalJun Campus, 1 Francis Crick Ave, Trumpington, Cambridge, CB2 0AA, UK

Ultrasound Image Segmentation Left Ventricular Ejection Fraction Estimation Echocardiography Self-Supervised Visual Representation Learning

Jun 19, 2026cs.CV

Radial Basis Function Networks as Projection Heads in Self-Supervised Learning

Self-supervised learning (SSL) typically relies on a backbone encoder followed by a small multilayer perceptron (MLP) projection head, which is conventionally discarded after training, while backbone quality is assessed via costly linear probing on labeled data. We argue that this approach including discarding the projector is rather computationally wasteful. Instead, we propose replacing the MLP head with a radial basis function network (RBFN), whose interpretable center and shape parameters can be exploited to judge representation quality without labels or a separate classifier. To this end, we introduce Scale-Normalized Separation (SNS), a novel label-free quality metric derived solely from the kernel centers and shapes learned during training. Across five canonical SSL architectures (MoCo, SimCLR, BYOL, SwAV and SimSiam) and four image classification datasets, we show that RBFN projection heads are competitive drop-in replacements for standard MLP projectors. We recommend constructing them with three RBF layers activated by the Gaussian radial basis function. Moreover, SNS exhibits strong to very strong positive correlation with established logistic regression metrics, demonstrating that a trained RBFN projector can act as a reliable proxy for backbone representation quality. We additionally publish a novel PyTorch compatible image classification dataset based on Google's Open Images V7 to facilitate reproducible research into representation learning.

Andreas Schliebitz, Heiko Tapken, Martin Atzmueller

Osnabrück University of Applied Sciences, Albrechtstr. 30, 49076 Osnabrück, Germany · Osnabrück University, Semantic Information Systems Group,2026 Wachsbleiche 27, 49090 Osnabrück Germany · German Research Center for Artificial Intelligence (DFKI), Hamburger Str. 24, 49084 Osnabrück, Germany

Representation Learning Self-Supervised Visual Representation Learning Radial Basis Function Networks

Jun 17, 2026cs.LG

3D-DLP: Self-Supervised 3D Object-Centric Scene Representation Learning

We introduce 3D-DLP, a self-supervised object-centric representation learning model that decomposes scene-level RGB-D or voxel observations into a set of 3D latent particles. Building on the Deep Latent Particles (DLP) framework, each particle encodes disentangled attributes, including 3D keypoint position, bounding box dimensions, and appearance features, and represents a distinct entity in the scene. The model learns interpretable per-particle segmentation maps through an end-to-end self-supervised reconstruction objective. We demonstrate on both simulated and real-world datasets that the learned latent space is interpretable and controllable: by manipulating particle positions and decoding, we can generate novel scene configurations. Furthermore, we show that leveraging these compact 3D latent particles for downstream robotic manipulation improves performance over baselines that either lack explicit 3D information or rely on memory-intensive dense 3D inputs without object-centric structure. Code and videos are available at https://eubooks3003.github.io/3d-dlp.

Ellina Zhang, Madhaven Iyengar, Amir Zadeh +4

Carnegie Mellon University · Lambda AI

Object-Centric Representation Learning 3D Scene Representation 3D Representation Learning Self-Supervised Visual Representation Learning

Jun 16, 2026cs.LG

Expanding SPHERE-JEPA: A Family of Statistical Regularizers for the Hypersphere

In Self-Supervised Learning (SSL), preventing representation collapse by explicitly enforcing a uniform distribution on the unit hypersphere has proven to be effective. However, current frameworks typically rely on sliced statistical regularizers such as SIGReg (used in LeJEPA) and SUSReg (used in SPHERE-JEPA), which approximate this continuous objective via Monte Carlo sampling along random 1D directions. This stochasticity injects projection variance into the training gradients, destabilizing optimization, and hindering convergence. In this work, we first show that analytically integrating out these random projections natively yields a deterministic Maximum Mean Discrepancy (MMD), bypassing the variance of sliced methods. Motivated by this equivalence, we formulate full-dimensional objectives for MMD, Kernel Stein Discrepancy (KSD), and Kullback-Leibler (KL) divergence directly on the sphere to enforce a uniform distribution. To prevent spatial bias, we equip these tests with rotationally invariant kernels constructed via spectral theory, systematically evaluating two canonical families: smooth exponential decay (Heat) and strict frequency cutoff (Bandlimited) filters. Empirically, removing projection-induced noise results in more stable optimization, faster convergence, and consistent improvements over stochastic sliced regularizers on ImageNet and Galaxy10. Furthermore, we reveal that the choice of the statistical test shapes the geometry of the learned latent space: MMD and KSD favor locally clustered organization suitable for object-centric domains, whereas the continuous KDE-based KL divergence promotes fine-grained instance separation, yielding the strongest results on unclustered procedural texture retrieval.

Léo Nicollier, Enric Meinhardt-Llopis, Max Dunitz +3

CB, ATT · Université Paris-Saclay, CNRS, ENS Paris-Saclay, Centre Borelli, France · CB +4

Maximum Mean Discrepancy Hyperspherical Representation Learning Self-Supervised Visual Representation Learning

Jun 14, 2026cs.CV

You Don't Need Strong Assumptions: Visual Representation Learning via Temporal Differences

Progress in AI has largely been driven by methods that assume less. As compute and data increase, approaches with weaker inductive biases generally outperform those with stronger assumptions. This is particularly characteristic of the field of Visual Representation Learning, where approaches have gone from being dominated by Supervised Learning, to Weakly Supervised Learning, to the now widespread success of Self-Supervised Learning without human labels. Yet, even modern Self-Supervised Learning approaches still depend on strong inductive biases such as augmentations, masking, or cropping. If this trend holds, even these remaining biases should become bottlenecks at scale -- and our experiments confirm this: the optimal strength of inductive biases decreases as data grows. This motivates the search for approaches that rely on fewer assumptions. To this end, we introduce Temporal Difference in Vision (TDV), a new paradigm for self-supervised learning from video that avoids existing inductive biases, relying instead on a causal assumption that the past causes the future. TDV functions by jointly training an image encoder and a motion encoder so that the current frame's representation plus the encoded motion equals the next frame's representation. Despite not leveraging any strong inductive biases, TDV matches state-of-the-art recipes on dense spatial tasks, laying the foundation for representation learning without strong assumptions.

Ninad Daithankar, Alexi Gladstone, Yann LeCun +1

1UIUC · 2New York University

Visual Representation Learning Video Representation Learning Self-Supervised Visual Representation Learning

Jun 12, 2026cs.CV

RATS! Patches Talk Through Registers: Emergent Parts in Register Attention Transformers

When humans see a bird, they recognize far more than just "bird" -- they see a head, wings, and talons, a structured assembly of reusable parts that can be identified across every bird they have ever seen. We ask whether a self-supervised visual model can discover the same compositional structure on its own. To this end, we propose RATS (Register Attention Transformers), which decomposes the classification token into N learnable register tokens that route patch information through an L->N->N->L bottleneck via a three-step compress-communicate-broadcast attention. The N registers are partitioned across the H attention heads, so that registers assigned to different heads do not interact with each other. Without auxiliary losses or part annotations, each register spontaneously specializes into a proto-semantic region whose emerging structure resembles object parts. RATS surpasses all baselines by +12 mIoU on average across five segmentation benchmarks, with consistent gains on ADE20K (+1.11 mIoU) and COCO (+0.2 AP^m). Its register dictionary further exhibits part-level consistency and semantic proximity across related categories. Our results suggest that RATS may provide a useful architectural prior for structured and interpretable visual representation learning.

Timing Yang, Predrag Neskovic, Jansen Seheult +4

1Johns Hopkins University · 2Office of Naval Research, Arlington, VA · Department of Laboratory Medicine and Pathology, Mayo Clinic,FLOPs (G)MN, USA

Visual Attention Visual Representation Learning Vision Transformer Object-Centric Representation Learning Compositional Representation Learning Semantic Segmentation+1

Jun 10, 2026cs.CV

Dual-State Slot Attention: Decoupling Appearance and Identity for Video Object-Centric Learning

Unsupervised video object-centric learning aims to decompose dynamic scenes into persistent, object-level representations without supervision. However, existing slot-based methods struggle to maintain stable object identity in challenging settings such as rapid motion and partial occlusion. First, they typically encode both the per-frame appearance of an object and its identity across frames in a single slot vector, creating an objective conflict that leads to slot swapping: reconstruction requires sensitivity to transient visual changes, whereas temporal consistency requires invariance to them. Second, the token renormalization used in Slot Attention can amplify weakly attending slots, allowing them to absorb tokens from other objects and destabilize slot-to-object correspondence. We propose Dual-State Slot Attention (DSSA), a fully self-supervised framework that addresses these limitations by separating appearance from identity and by reducing spurious updates from weakly matching slots. DSSA decomposes each slot into a local state for per-frame appearance and an identity state for temporally stable object information, thereby aligning reconstruction and temporal consistency with separate representations. The identity state is updated through a learned recurrent transition that acts as a temporal filter on the local state, while competition-modulated aggregation (CMA) down-weights updates from weakly matching slots and prevents them from absorbing tokens from other objects. Experiments on MOVi-C, MOVi-D, and YouTube-VIS demonstrate that DSSA consistently improves segmentation quality and temporal consistency over prior methods, while also yielding stronger downstream object recognition and video dynamics prediction. Code and models will be made publicly available upon acceptance.

Sieu Tran, Duc Nguyen, Hao Vo +2

University of Arkansas Fayetteville, AR, USA

Object-Centric Representation Learning Slot Attention Self-Supervised Visual Representation Learning

Jun 6, 2026cs.CV

SMI: Efficient Self-Supervised Learning via Mutual-Information-Inspired Dependency Optimization

Self-supervised learning (SSL) has achieved remarkable representation learning performance, but many existing methods rely on large batch sizes, memory banks, momentum encoders, or global synchronization mechanisms that substantially increase computational cost and training complexity. In this work, we propose Semantic Mutual Information (SMI), a lightweight self-supervised objective derived from a mutual-information-inspired dependency formulation under Gaussian assumptions. Unlike conventional correlation matching objectives that operate on high-dimensional feature correlation matrices, SMI performs optimization on a sample-level dependency matrix through a nonlinear transformation of pairwise correlations. This formulation induces distinct optimization dynamics that emphasize strongly dependent semantic pairs while maintaining representation diversity. Experimental results on ImageNet using a ResNet-50 backbone demonstrate that SMI achieves competitive linear evaluation performance relative to state-of-the-art SSL approaches while substantially reducing computational complexity. Across multiple low-resource benchmarks, SMI consistently improves transfer performance over Barlow Twins, particularly on fine-grained datasets. Furthermore, analyses of optimization dynamics and representation geometry suggest improved alignment--redundancy balance, greater feature diversity, and more spatially localized semantic representations. These results indicate that nonlinear dependency optimization provides an effective and computationally efficient alternative to conventional correlation-based self-supervised learning objectives.

Pritam Mishra, Coloma Ballester, Dimosthenis Karatzas

Universitat Pompeu Fabra, Barcelona, Spain · Universitat Autònoma de Barcelona, Barcelona, Spain

Self-Supervised Learning Self-Supervised Visual Representation Learning

Jun 5, 2026eess.IV

DaX: Learning General Pathology Representations Across Scales

Computational pathology requires visual representations that transfer across diverse clinical endpoints and remain robust to variation in magnification, staining, scanner type, slide preparation, and input resolution. We present DaX, a pathology vision foundation model that adapts DINOv3-style self-supervised learning to whole-slide histopathology. DaX is initialized from natural-image DINOv3 weights and incorporates continuous magnification training, cross-scale tissue views, orientation-agnostic and acquisition-robust augmentation, multi-input-size training, and Gram-anchored dense consistency. These designs aim to connect local cellular morphology with global tissue architecture while stabilizing dense token-level representations across input scales. We further construct a WSI-level benchmark comprising 161 clinically meaningful tasks from 44 public datasets, covering 28,182 patients and 34,394 slides across four clinical domains and nine task categories. All models are evaluated under a fixed patient-level cross-validation protocol with fold-level statistical ranking, enabling reproducible comparisons that are less sensitive to split-dependent variation. Across this benchmark, DaX achieves the highest mean performance across tasks and consistently strong task-level ranking scores, with gains spanning diagnostic pathology, biomarker and molecular profiling, tissue/specimen context, and risk, response, and prognosis. These results support DaX as a transferable visual encoder for computational pathology and provide a standardized evaluation framework for future pathology foundation models. Project page: https://alibaba-damo-academy.github.io/DaX/benchboard/.

Bokai Zhao, Yiyang Zhang, Long Bai +3

DAMO Academy, Alibaba Group · Institute of Automation, Chinese Academy of Sciences · Hupan Lab

Pathology Foundation Models Medical Image Representation Learning Medical Image Benchmarks Whole-Slide Image Analysis Self-Supervised Visual Representation Learning

Jun 4, 2026cs.LG

Balancing Image Compression and Generation with Bootstrapped Tokenization

Despite progress in image tokenization, standard methods encode redundant information by mixing all granularities within each token, thus redundancy persists between tokens. The mix of information of different granularity also complicates the training of generators. This paper introduces SelfBootTok, a method that resolves this by cleanly decomposing information into global and local token groups. Through self-bootstrapped learning, the model predicts local details exclusively from global tokens, shifting the burden of visual details from the generator to the tokenizer. Consequently, our generator is far more efficient, requiring only global tokens and reducing computation by approximately 40%, while delivering superior reconstruction and generation. Moreover, this paradigm scales elegantly: by leveraging more data or parameters to self-supervise local representation learning, SelfBootTok achieves a new state-of-the-art gFID score of 1.56 using only 64 tokens.

Haozhe Chi, Jinghan Li, Hao Jiang +4

1Peking University · 2Central Media Technology Institute, Huawei

Learned Image Compression Image Tokenization Visual Tokenization Autoregressive Image Generation Self-Supervised Visual Representation Learning Visual Token Compression

Jun 3, 2026cs.CV

Who Needs Labels? Adapting Vision Foundation Models With the Metadata You Already Have

We propose a label-free approach to adapt powerful but generic vision foundation models to specialized scientific domains. Standard supervised fine-tuning is often ill-suited to these settings: labels are scarce, and task-specific training can collapse the model's generality and hurt robustness. We instead leverage metadata to adapt representations to new domains in a self-supervised manner. Our method, FINO, combines a standard self-supervised objective with flexible metadata guidance that handles both highly granular discrete metadata and continuous metadata. It encourages the representation to preserve informative factors while suppressing spurious ones. Across subcellular fluorescence microscopy, Earth observation, wildlife monitoring, and medical imaging, FINO consistently outperforms standard unsupervised domain adaptation and fully supervised adaptation. It also exceeds highly-specialized domain-specific state of the art, while using no task labels for backbone adaptation and only lightweight probes for supervision.

Elouan Gardès, Seung Eun Yi, Kartik Ahuja +6

Meta FAIR, Paris · Columbia University, New York · LIGM, CNRS, Gustave Eiffel, ENPC, IP Paris

Unsupervised Domain Adaptation Domain Adaptation Vision Foundation Model Adaptation Self-Supervised Visual Representation Learning

Jun 2, 2026cs.CV

CoralBay: A Self-Supervised CT Foundation Model

Self-supervised learning has enabled large-scale pre-training on 2D natural images, producing general-purpose visual representations that transfer effectively across tasks. However, many medical imaging modalities, such as CT scans, are inherently three-dimensional and differ fundamentally from natural images in both structure and semantics. Volumetric modalities capture spatial continuity, organ anatomy, and intensity-based tissue properties (e.g., Hounsfield Units), which are not adequately modeled by 2D pre-training. To bridge this gap, we introduce CoralBay, a self-distillation framework that extends DINO by using a hierarchical 3D Swin backbone and applying self-distillation to concatenated multi-scale features, enabling data-efficient self-supervised learning of rich spatial representations that encode both global semantics and fine-grained local structure. As a result, CoralBay transfers effectively to a wide range of downstream radiological tasks, demonstrating strong and consistent performance across diverse anatomical targets. In addition, we contribute to the open-source \eva framework by introducing a public, reproducible 3D radiology leaderboard that unifies multiple datasets and establishes a standardized benchmark for evaluating volumetric representation learning methods.

Ioannis Gatopoulos, Nicolas Känzig, Sebastian Otálora +1

Medical Imaging Foundation Models 3D Representation Learning Self-Distillation Computed Tomography Medical Image Benchmarks Self-Supervised Visual Representation Learning

Jun 2, 2026cs.CV

$A^2$ : Smaller Self-Supervised ViTs Localize Better than Larger Ones

Robust visual classification often depends on localizing the main foreground objects in an image while ignoring contextual distractors. Surprisingly, we find that the attention maps of smaller self-supervised ViTs localize foreground objects better than those of larger ViTs. However, we still need large ViTs, because they extract richer representations from each patch. To get the best of both worlds, good localization and rich representations, we propose

A^2

, a simple method that leverages this inverse scaling finding by decoupling where to look (a small attention model) from what to extract (a large embedding model): we crop around the attention peaks of a small model and embed the crops with a larger model.

A^2

uses entirely pretrained features, requires no group labels, and does not require per-dataset attention or backbone training. Across 5 benchmarks,

A^2

is competitive with backbone-matched loss-level methods like DFR, and outperforms end-to-end attention training under stronger distribution shifts.

Sreehari Rammohan, Huy Ha, Carl Vondrick

a Columbia University · b Stanford University

Visual Attention Vision Transformer Self-Supervised Visual Representation Learning