Invariant Representation Learning

Follow this topic Ask about this topic

Momentum

21 papers in the last four weeks, up 110% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 153

May 7, 2026cs.LG

Invariant Features in Language Models: Geometric Characterization and Model Attribution

Language models exhibit strong robustness to paraphrasing, suggesting that semantic information may be encoded through stable internal representations, yet the structure and origin of such invariance remain unclear. We propose a local geometric framework in which semantically equivalent inputs occupy structured regions in latent space, with paraphrastic variation along nuisance directions and semantic identity preserved in invariant subspaces. Building on this view, we make three contributions: (1) a geometric characterization of invariant latent features, (2) a contrastive subspace discovery method that separates semantic-changing from semantic-preserving variation, and (3) an application of invariant representations to zero-shot model attribution. Across models and layers, empirical results support these contributions. Invariant structure emerges in specific depth regions, semantic displacement lies largely outside the nuisance subspace, and representation-level interventions indicate a causal role of invariant components in model outputs. Invariant representations also capture model-specific geometric patterns, enabling accurate attribution. These findings suggest that semantic invariance can be viewed as a local geometric property of latent representations, offering a principled perspective on how language models organize meaning.

Agnibh Dasgupta, Abdullah Tanvir, Xin Zhong

Department of Computer Science University of Nebraska Omaha, Omaha, NE, USA

Neural Representation Geometry Invariant Representation Learning Representation Geometry in Language Models

May 7, 2026cs.CV

Adaptive Physical-Facial Representation Fusion via Subject-Invariant Cross-Modal Prompt Tuning for Video-Based Emotion Recognition

Emotion recognition from facial videos enables non-contact inference of human emotional states. Although facial expressions are widely used cues, they cannot fully reflect intrinsic affective states. Remote photoplethysmography (rPPG) provides complementary physiological information, but it is highly susceptible to noise and inter-subject variability, limiting generalization to unseen individuals. Existing multimodal methods combine facial and rPPG features, yet their fusion strategies often disrupt pretrained facial representations and lack explicit mechanisms to suppress subject-specific variations. To address these issues, we propose a subject-invariant cross-modal prompt-tuning framework for video-based emotion recognition. Specifically, rPPG waveforms are transformed into noise-robust time-frequency representations (TFRs), from which modality-complementary prompts are generated to modulate facial tokens within a frozen Vision Transformer (ViT). This design enables effective cross-modal interaction while preserving the generalizable facial representations learned by the pretrained backbone. In addition, we introduce a decoupled shared-specific adapter (DSSA) into each ViT layer to explicitly separate subject-shared and subject-specific components, thereby improving cross-subject generalization. Experiments on the MAHNOB-HCI and DEAP benchmarks demonstrate that the proposed method consistently outperforms strong baselines in both recognition accuracy and generalization ability, highlighting its effectiveness for video-based emotion recognition.

Xiwen Luo, Jia Li, Rencheng Song +2

Department of Biomedical Engineering, and Anhui Province Key Laboratory of Measuring Theory and Precision Instrument, Hefei University of Technology, Hefei, 230009 China · School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, 230601 China

Multimodal Emotion Recognition Invariant Representation Learning Multimodal Fusion Emotion Recognition

May 6, 2026cs.CV

CPCANet: Deep Unfolding Common Principal Component Analysis for Domain Generalization

Domain Generalization (DG) aims to learn representations that remain robust under out-of-distribution (OOD) shifts and generalize effectively to unseen target domains. While recent invariant learning strategies and architectural advances have achieved strong performance, explicitly discovering a structured domain-invariant subspace through second-order statistics remains underexplored. In this work, we propose CPCANet, a novel framework grounded in Common Principal Component Analysis (CPCA), which unrolls the iterative Flury-Gautschi (FG) algorithm into fully differentiable neural layers. This approach integrates the statistical properties of CPCA into an end-to-end trainable framework, enforcing the discovery of a shared subspace across diverse domains while preserving interpretability. Experiments on four standard DG benchmarks demonstrate that CPCANet achieves state-of-the-art (SOTA) performance in zero-shot transfer. Moreover, CPCANet is architecture-agnostic and requires no dataset-specific tuning, providing a simple and efficient approach to learning robust representations under distribution shift. Code is available at https://github.com/wish44165/CPCANet.

Yu-Hsi Chen, Abd-Krim Seghouane

The University of Melbourne

Principal Component Analysis Domain Generalization OOD Generalization Invariant Representation Learning Deep Unfolding

May 6, 2026cs.LG

Bridging Input Feature Spaces Towards Graph Foundation Models

Unlike vision and language domains, graph learning lacks a shared input space, as input features differ across graph datasets not only in semantics, but also in value ranges and dimensionality. This misalignment prevents graph models from generalizing across datasets, limiting their use as foundation models. In this work, we propose ALL-IN, a simple and theoretically grounded method that enables transferability across datasets with different input features. Our approach projects node features into a shared random space and constructs representations via covariance-based statistics, thus eliminating dependence on the original feature space. We show that the computed node-covariance operators and the resulting node representations are invariant in distribution to permutations of the input features. We further demonstrate that the expected operator exhibits invariance to general orthogonal transformations of the input features. Empirically, ALL-IN achieves strong performance across diverse node- and graph-level tasks on unseen datasets with new input features, without requiring architecture changes or retraining. These results point to a promising direction for input-agnostic, transferable graph models.

Moshe Eliasof, Krishna Sri Ipsit Mantri, Beatrice Bevilacqua +2

University of Cambridge · Ben-Gurion University of the Negev · Purdue University

Graph Representation Learning Transfer Learning Graph Foundation Models Invariant Representation Learning

May 6, 2026cs.LG

Unsat Core Prediction through Polarity-Aware Representation Learning over Clause-Literal Hypergraphs

Graph neural networks have been widely used in Boolean satisfiability (SAT) tasks to learn structural information from SAT formulas. The goal of these studies is to solve SAT instances or to enhance SAT solvers, including tasks such as unsat-core prediction. However, most existing approaches model a SAT formula as a bipartite graph or a directed acyclic graph, which are less direct in capturing clause-level and higher-order interactions among literals and clauses. Moreover, these approaches are limited in modeling intrinsic polarity-related properties of SAT, such as the complementary relationship between the positive and negative literals of a variable. To address these limitations, we propose a polarity-aware representation learning framework over clause-literal hypergraphs. We model SAT formulas as clause-literal hypergraphs augmented with a clause incidence graph to capture higher-order structural interactions. We then introduce a polarity-aware decomposition mechanism that separates variable representations into polarity invariant and equivariant components, explicitly modeling the relationship between positive and negative literals, with the resulting literal representations propagated along the hypergraph structure. We further incorporate a polarity-inversion consistency regularization to reinforce polarity-consistent representations during training. Experimental results on multiple SAT datasets demonstrate the effectiveness of the proposed approach.

Zhenchao Sun, Shuai Ma, Ping Lu +1

SKLCCSE Lab, Beihang University, Beijing, China.

Hypergraph Neural Networks Constraint Satisfaction Invariant Representation Learning

May 2, 2026eess.SP

MU-SHOT-Fi: Self-Supervised Multi-User Wi-Fi Sensing with Source-free Unsupervised Domain Adaptation

Deep learning has been widely adopted for WiFi CSI-based human activity recognition (HAR) due to its ability to learn spatio-temporal features in a privacy-preserving and cost-effective manner. However, DL-based models generalize poorly across environments, a challenge amplified in multi-user settings where overlapping activities cause CSI entanglement and domain shifts. Practical deployments often limit access to labeled source data due to privacy constraints, motivating source-free adaptation using only unlabeled target-domain CSI and a pre-trained source model. In this paper, we propose MU-SHOT-Fi, a source-free unsupervised domain adaptation framework for single- and multi-user Wi-Fi sensing. MU-SHOT-Fi employs permutation-invariant set prediction with Hungarian matching during source training, followed by frozen-classifier backbone adaptation in the target domain. To enable stable adaptation without labels, we introduce occupancy-weighted information maximization that prevents model collapse by focusing diversity regularization on likely-occupied slots while excluding the dominant class from marginal entropy. Additionally, we employ binary rotation prediction as spatial self-supervision that exploits CSI frequency-time structure to learn domain-invariant features. For single-user scenarios, we introduce SU-SHOT-Fi by replacing occupancy weighting with standard information maximization and incorporating contrastive predictive coding to exploit temporal consistency. Extensive experiments on the WiMANS and Widar 3.0 datasets across cross-environment, cross-frequency, cross-orientation, and combined domain shifts demonstrate that MU-SHOT-Fi effectively recovers multi-user exact-activity classification performance under large domain shifts while maintaining accurate occupancy estimation and preventing collapse toward dominant classes.

Ahmed Y. Radwan, Hina Tabassum

department of Electrical Engineering and Computer Science, York University, Toronto, ON M3J 1P3, Canada

Source-Free Domain Adaptation Human Activity Recognition Self-Supervised Learning Permutation-Invariant Learning Invariant Representation Learning Wi-Fi Sensing

Apr 30, 2026cs.CV

Parameter-Efficient Architectural Modifications for Translation-Invariant CNNs

Convolutional Neural Networks (CNNs) are widely assumed to be translation-invariant, yet standard architectures exhibit a startling fragility: even a single-pixel shift can drastically degrade performance due to their reliance on spatially dependent fully connected layers. In this work, we resolve this vulnerability by proposing a lightweight 'Online Architecture' strategy. By strategically inserting Global Average Pooling (GAP) layers at various network depths, we effectively decouple feature recognition from spatial location. Using VGG-16 as a primary case study, we demonstrate that this architectural modification achieves a massive 98% reduction in trainable parameters (from 5.2M to just 82K) and a 90% reduction in total network size (138M to 14M). Despite this drastic pruning, our variants maintain competitive Top-1 accuracy on ImageNet (66.4%) while doubling translational robustness, reducing average relative loss from 0.09 to 0.05. Furthermore, our analysis identifies a fundamental limit to invariance: while GAP resolves macroscopic sensitivity, discrete pooling operations introduce a residual periodic aliasing that prevents perfect pixel-level stability. Finally, we extend these findings to Perceptual Image Quality Assessment (IQA) by integrating our invariant backbones into the LPIPS framework. The resulting metric significantly outperforms the retrained baseline in generalization across the KADID-10k dataset (Spearman 0.89 vs. 0.75) and achieves a near-perfect alignment with human psychophysical response curves on the RAID dataset (Spearman 0.95). These results confirm that enforcing architectural invariance is a far more efficient and biologically plausible path to robustness than traditional data augmentation. Data and code are publicly available. The data and code are publicly available to facilitate validation and further research.

Nuria Alabau-Bosque, Jorge Vila-Tomas, Paula Dauden-Oliver +2

Image Processing Lab, Universitat de València, Carrer del Catedr´atic Jos´e Beltr´an Martinez, Paterna, 46980, Spain.

Neural Network Robustness Convolutional Neural Networks Invariant Representation Learning Neural Network Compression

Apr 29, 2026cs.CV

Leveraging Imperfect Medical Data: A Manifold-Consistent Spatio-Temporal Network for Sensor-based Human Activity Recognition

Sensor-based Human Activity Recognition (HAR) has attracted increasing attention in medical and healthcare monitoring, particularly with the growth of Internet of Medical Things (IoMT). However, in real-world wearable sensing scenarios, IoMT signals are often corrupted by missing measurements, sensor failures, and environmental noise, which significantly degrade the performance of conventional deep learning models that assume clean and complete inputs. To address this challenge, we propose a Manifold-Consistent Spatio-Temporal Network (MCSTN) for robust HAR under imperfect sensing conditions. The proposed framework introduces a dual-level corruption modeling mechanism that simulates realistic sensor imperfections through both physical-level corruption and diffusion-driven continuous corruption. By enforcing representation consistency across multiple corrupted views, the model learns stable and corruption-invariant semantic representations. Furthermore, we design a dual-stream spatio-temporal architecture that explicitly decouples temporal dynamics modeling and spatial correlation learning. The temporal stream captures long-term activity dynamics, while the spatial stream models inter-sensor relationships, enabling more effective spatio-temporal representation learning. Extensive experiments on three widely used HAR benchmark datasets, PAMAP2, Opportunity, and WISDM, demonstrate that the proposed MCSTN achieves competitive performance compared with existing state-of-the-art methods, particularly under imperfect sensing conditions. These results validate the effectiveness and robustness of the proposed framework for real-world wearable IoMT sensing applications.

Jiangtao Fan, Anish Jindal, Amir Atapour-Abarghouei

Department of Computer Science, Durham University, Durham, DH1 3LE, UK

Human Activity Recognition Wearable Sensing Invariant Representation Learning Time Series Representation Learning

Apr 28, 2026stat.ML

Robust Representation Learning through Explicit Environment Modeling

We consider learning from labeled data collected across multiple environments, where the data distribution may vary across these environments. This problem is commonly approached from a causal perspective, seeking invariant representations that retain causal factors while discarding spurious ones. However, this framework assumes that the environment has no direct effect on the target. In contrast, we consider settings in which this assumption fails, but still aim to learn representations that support robust prediction on average across previously unseen environments. To this end, we study representations learned by explicitly modeling variation across environments and then marginalizing that variation out. We analyze the resulting representations and characterize when they are preferable to those learned by causal invariant-representation methods. We propose a concrete method based on generalized random-intercept models, a class of predictors in which such marginalization is possible, and study their generalization properties. Empirically, we show that these models outperform invariant-learning methods across a range of challenging settings.

Yuli Slavutsky, David M. Blei

Department of Statistics Columbia University · Departments of Statistics and Computer Science Columbia University

Representation Learning Domain Generalization Invariant Representation Learning

Apr 27, 2026cs.CV

ShapeY: A Principled Framework for Measuring Shape Recognition Capacity via Nearest-Neighbor Matching

Object recognition (OR) in humans relies heavily on shape cues and the ability to recognize objects across varying 3D viewpoints. Unlike humans, deep networks often rely on non-shape cues such as texture and background, leading to vulnerabilities in generalization and robustness. To address this gap, we introduce ShapeY, a novel and principled benchmarking framework designed to evaluate shape-based recognition capability in OR systems. ShapeY comprises 68,200 grayscale images of 200 3D objects rendered from multiple viewpoints and optionally subjected to non-shape ``appearance'' changes. Using a nearest-neighbor matching task, ShapeY specifically probes the fine-grained structure of an OR system's embedding space by evaluating whether object views are clustered by 3D shape similarity across varying 3D viewpoints and other non-shape changes. ShapeY provides a suite of quantitative and qualitative performance readouts, including error rate graphs, viewpoint tuning curves, histograms of positive and negative matching scores, and grids showing ordered best matches, which together offer a comprehensive evaluation of an OR system's shape understanding capability. Testing of 321 pre-trained networks with diverse architectures reveals significant challenges in achieving robust shape-based recognition: even state-of-the-art models struggle to generalize consistently across 3D viewpoint and appearance changes, and are prone to infrequent but egregious matches of objects of obviously completely different shape. ShapeY establishes a principled framework for advancing artificial vision systems toward human-like shape recognition capabilities, emphasizing the importance of disentangled and invariant object encodings.

Jong Woo Nam, Amanda S. Rios, Bartlett W. Mel

Invariant Representation Learning Fine-Grained Visual Perception

Apr 27, 2026cs.CV

Probing CLIP's Comprehension of 360-Degree Textual and Visual Semantics

The dream of instantly creating rich 360-degree panoramic worlds from text is rapidly becoming a reality, yet a crucial gap exists in our ability to reliably evaluate their semantic alignment. Contrastive Language-Image Pre-training (CLIP) models, standard AI evaluators, predominantly trained on perspective image-text pairs, face an open question regarding their understanding of the unique characteristics of 360-degree panoramic image-text pairs. This paper addresses this gap by first introducing two concepts: \emph{360-degree textual semantics}, semantic information conveyed by explicit format identifiers, and \emph{360-degree visual semantics}, invariant semantics under horizontal circular shifts. To probe CLIP's comprehension of these semantics, we then propose novel evaluation methodologies using keyword manipulation and horizontal circular shifts of varying magnitudes. Rigorous statistical analyses across popular CLIP configurations reveal that: (1) CLIP models effectively leverage explicit textual identifiers, demonstrating an understanding of 360-degree textual semantics; and (2) CLIP models fail to robustly preserve semantic alignment under horizontal circular shifts, indicating limited comprehension of 360-degree visual semantics. To address this limitation, we propose a LoRA-based fine-tuning framework that explicitly instills invariance to circular shifts. Our fine-tuned models exhibit improved comprehension of 360-degree visual semantics, though with a slight degradation in original semantic evaluation performance, highlighting a fundamental trade-off in adapting CLIP to 360-degree panoramic images. Code is available at https://github.com/littlewhitesea/360Semantics.

Hai Wang, Xiaochen Yang, Mingzhi Dong +1

Department of Statistical Science, University College London, UK · School of Mathematics and Statistics, University of Glasgow, UK · Department of Computer Science, University of Bath, UK.

VLM Evaluation VLM Adaptation Text-Image Alignment Invariant Representation Learning Vision-Language Pretraining

Apr 27, 2026cs.CL

Improving Robustness of Tabular Retrieval via Representational Stability

Transformer-based table retrieval systems flatten structured tables into token sequences, making retrieval sensitive to the choice of serialization even when table semantics remain unchanged. We show that semantically equivalent serializations, such as

\texttt{csv}

,

\texttt{tsv}

,

\texttt{html}

,

\texttt{markdown}

, and

\texttt{ddl}

, can produce substantially different embeddings and retrieval results across multiple benchmarks and retriever families. To address this instability, we treat serialization embedding as noisy views of a shared semantic signal and use its centroid as a canonical target representation. We show that centroid averaging suppresses format-specific variation and can recover the semantic content common to different serializations when format-induced shifts differ across tables. Empirically, centroid representations outrank individual formats in aggregate pairwise comparisons across

\texttt{MPNet}

,

\texttt{BGE-M3}

,

\texttt{ReasonIR}

, and

\texttt{SPLADE}

. We further introduce a lightweight residual bottleneck adapter on top of a frozen encoder that maps single-serialization embeddings towards centroid targets while preserving variance and enforcing covariance regularization. The adapter improves robustness for several dense retrievers, though gains are model-dependent and weaker for sparse lexical retrieval. These results identify serialization sensitivity as a major source of retrieval variance and show the promise of post hoc geometric correction for serialization-invariant table retrieval.

Kushal Raj Bhandari, Adarsh Singh, Jianxi Gao +2

Department of Computer Science Arizona State University Tempe, Arizona, USA · ScaleAI San Francisco, CA, USA

Retrieval Robustness Information Retrieval Table Retrieval Invariant Representation Learning Dense Retrieval

Apr 24, 2026cs.RO

Magnetic Indoor Localization through CNN Regression and Rotation Invariance

Indoor positioning is an essential technology for a wide range of applications in GNSS-denied environments, including indoor navigation and IoT systems. Combining convolutional neural networks (CNNs) and magnetic field-based features offers a low-cost, infrastructure-free solution for precise positioning. While magnetic fingerprints are a promising approach for indoor positioning, models trained on raw 3D magnetometer data are highly sensitive to device orientation. We address this by using two rotation invariant features derived from the 3D magnetic field: the norm (Mn) and the projection onto the gravity axis (Mg). We train a lightweight 7-layer dilated CNN (MagNetS/XL) on magnetic sequences to directly regress (x, y) positions. Using the MagPie dataset (three buildings, handheld trajectories), we systematically evaluate fixed and random rotations of test and/or train data. Raw 3D inputs (Mx, My , Mz) exhibit isotropic error increases under fixed 90° rotations and further degrade with growing random rotations. In contrast, 2D (Mn, Mg) inputs maintain rotation invariant accuracy and surpass the 3D inputs once rotation exceeds building-specific thresholds for three reference buildings: 0° for Loomis (large), 5° for Talbot (medium), and 6° for CSL (small). MagNetXL achieves or exceeds state-of-the-art accuracy on the MagPie dataset, and MagNetS delivers similar performance with roughly one third of the parameters, favoring mobile deployment. These results show that the robustness gained from rotation invariant inputs outweighs the loss of input dimensionality in realistic usage, allowing mapping and localization without orientation alignment or added infrastructure.

Helge Rosé, Konstantin Klipp, Tom Koubek +2

Fraunhofer Institute for Open Communication Systems (FOKUS) Berlin, Germany · Daimler Center for Automotive Information Technology Innovations (DCAITI) Technical University Berlin Berlin, Germany

GNSS-Denied Localization Convolutional Neural Networks Invariant Representation Learning

Apr 21, 2026cs.LG

Disentangling Damage from Operational Variability: A Label-Free Self-Supervised Representation Learning Framework for Output-Only Structural Damage Identification

Damage identification is a core task in structural health monitoring. In practice, however, its reliability is often compromised by confounding non-damage effects, such as variations in excitation and environmental conditions, which can induce changes comparable to or larger than those caused by structural damage. To address this challenge, this study proposes a self-supervised label-free disentangled representation learning framework for robust vibration-based structural damage identification. The proposed framework employs an autoencoder with two latent representations to learn directly from raw vibration acceleration signals. A self-supervised invariance regularization, implemented via Variance-Invariance-Covariance Regularization (VICReg), is imposed on one latent representation using baseline data where structural damage is assumed constant but operational and environmental conditions vary. In addition, a frequency-domain constraint is introduced to enforce agreement between the power spectral density reconstructed from the latent representation and that computed from the corresponding input time series. Together, these mechanisms promote disentanglement, enabling the learned representation to be sensitive to damage-related characteristics while remaining invariant to nuisance variability. The framework is trained in a fully end-to-end and label-free manner, requiring no prior information on damage, excitation, or environmental conditions, making it well-suited for real-world applications. Its effectiveness is validated on two distinct real-world vibration datasets, including a bridge and a gearbox. The results demonstrate robustness to operational variability, strong generalization capability, and good performance in both damage detection and quantification.

Xudong Jian, Charikleia Stoura, Simon Scandella +1

Department of Civil, Environmental and Geomatic Engineering, ETH Zürich, Zürich, Switzerland. · Department of Mechanical Engineering, Politecnico di Milano, Italy

Disentangled Representation Learning Self-Supervised Time Series Representation Learning Invariant Representation Learning Structural Health Monitoring

Apr 20, 2026cs.LG

Learning Invariant Modality Representation for Robust Multimodal Learning from a Causal Inference Perspective

Multimodal affective computing aims to predict humans' sentiment, emotion, intention, and opinion using language, acoustic, and visual modalities. However, current models often learn spurious correlations that harm generalization under distribution shifts or noisy modalities. To address this, we propose a causal modality-invariant representation (CmIR) learning framework for robust multimodal learning. At its core, we introduce a theoretically grounded disentanglement method that separates each modality into causal invariant representation' and environment-specific spurious representation' from a causal inference perspective. CmIR ensures that the learned invariant representations retain stable predictive relationships with labels across different environments while preserving sufficient information from the raw inputs via invariance constraint, mutual information constraint, and reconstruction constraint. Experiments across multiple multimodal benchmarks demonstrate that CmIR achieves state-of-the-art performance. CmIR particularly excels on out-of-distribution data and noisy data, confirming its robustness and generalizability.

Sijie Mai, Shiqin Han

School of Computer Science, South China Normal University

Disentangled Representation Learning Causal Representation Learning Multimodal Robustness Multimodal Disentangled Representation Learning Sentiment Analysis Multimodal Emotion Recognition+2

Mar 31, 2026stat.ML

Breaking Data Symmetry is Needed For Generalization in Feature Learning Kernels

Grokking occurs when a model achieves high training accuracy but generalization to unseen test points happens long after that. This phenomenon was initially observed on a class of algebraic problems, such as learning modular arithmetic (Power et al., 2022). We study grokking on algebraic tasks in a class of feature learning kernels via the Recursive Feature Machine (RFM) algorithm (Radhakrishnan et al., 2024), which iteratively updates feature matrices through the Average Gradient Outer Product (AGOP) of an estimator in order to learn task-relevant features. Our main experimental finding is that generalization occurs only when a certain symmetry in the training set is broken. Furthermore, we empirically show that RFM generalizes by recovering the underlying invariance group action inherent in the data. We find that the learned feature matrices encode specific elements of the invariance group, explaining the dependence of generalization on symmetry.

Marcel Tomàs Bernal, Neil Rohit Mallinar, Mikhail Belkin

CFIS UPC · UC San Diego

Symmetry Breaking Kernel Methods Invariant Representation Learning Grokking

Mar 29, 2026cs.LG

On the Asymptotics of Self-Supervised Pre-training: Two-Stage M-Estimation and Representation Symmetry

Self-supervised pre-training, where large corpora of unlabeled data are used to learn representations for downstream fine-tuning, has become a cornerstone of modern machine learning. While a growing body of theoretical work has begun to analyze this paradigm, existing bounds leave open the question of how sharp the current rates are, and whether they accurately capture the complex interaction between pre-training and fine-tuning. In this paper, we address this gap by developing an asymptotic theory of pre-training via two-stage M-estimation. A key challenge is that the pre-training estimator is often identifiable only up to a group symmetry, a feature common in representation learning that requires careful treatment. We address this issue using tools from Riemannian geometry to study the intrinsic parameters of the pre-training representation, which we link with the downstream predictor through a notion of orbit-invariance, precisely characterizing the limiting distribution of the downstream test risk. We apply our main result to several case studies, including spectral pre-training, factor models, and Gaussian mixture models, and obtain substantial improvements in problem-specific factors over prior art when applicable.

Mohammad Tinati, Stephen Tu

Department of Electrical and Computer Engineering, University of Southern California

Self-Supervised Pre-Training Invariant Representation Learning Statistical Learning Theory

Mar 26, 2026cs.AI

When Sensing Varies with Contexts: Context Probing for Tactile Few-Shot Class-Incremental Learning

Few-shot class-incremental learning (FSCIL) aims to recognize novel classes from only a few labeled samples while retaining previously learned knowledge. Although recent FSCIL methods have achieved substantial progress on visual benchmarks, they remain limited in tactile sensing, where the same material may produce markedly different observations under different acquisition contexts, such as sensing devices, contact states, scanning trajectories, and interaction conditions. In tactile FSCIL, the challenges of few-shot learning and class-incremental learning are further amplified by acquisition context: the limited support samples may not only be scarce, but also carry context-induced biases. Once the resulting biased prototypes are inserted into the classifier, they may affect the decision boundaries in subsequent sessions. To address this problem, we propose Context-Probing Few-Shot Class-Incremental Learning (CoP-FSCIL), a context-aware framework for tactile FSCIL. CoP-FSCIL first employs Context-Probing Intervention (CPI) to diagnose local context-sensitive variations in tactile representations. It then introduces a Probe-Conditioned Quotient Adapter (PCQA) to suppress context-sensitive components identified by the probes. Finally, Probe-Stability Prototype Calibration (PSPC) estimates support sample reliability from probe-induced embedding fluctuations and calibrates stochastic prototypes accordingly. Experiments on HapTex and LMT108 show that CoP-FSCIL consistently outperforms representative FSCIL baselines, and extended experiments on audio FSCIL further demonstrate the generality of the proposed context probing mechanism. The source code is currently being prepared and will be released soon.

Yifeng Lin, Aiping Huang, Wenxi Liu +4

Fujian Key Laboratory for Intelligent Processing and Wireless Transmission of Media Information, College of Physics and Information Engineering, Fuzhou University, Fuzhou 350108, China

Class-Incremental Learning Prototype-Based Classification Tactile Sensing Few-Shot Learning Invariant Representation Learning

Mar 5, 2026cs.RO

TransMASK: Masked State Representation through Learned Transformation

When humans learn new manipulation skills, they are able to generalize these skills to new contexts and environments. In particular, when learning, humans can easily separate task-relevant aspects of the environment (e.g., object location) and distractors (e.g., the table color). Ideally, robot policies should be able to learn similar, generalizable representations, but instead they often fail under small environment shifts such as changes in lighting, object instance, or initial configuration. In this paper, we propose a self-supervised method that learns a mask which, when multiplied by the observed image features, attempts to transform these features to retain only those which are relevant to the task. Our method --- which we call TransMASK --- can be combined with a variety of imitation learning frameworks (such as diffusion policies) without any additional labels or alterations to the loss function. By introducing a learned mask to the network during training, we aim to induce competitive pressure among the image features during training to force the policy to only attend to features which are consistently task-relevant. We find empirically that our masks are interpretable and can reject known spurious features such as the position of distractor objects or the background color. When compared to other representation learning methods for imitation learning, we find that TransMASK results in policies that are more robust to distribution shifts for irrelevant features, achieving at least 30 % improvement over the baselines when tested on out-of-distribution environments. See our project website: https://transmask.github.io/TransMASK/

Sagar Parekh, Preston Culbertson, Dylan P. Losey

Mechanical Engineering Department, Virginia Tech · Computer Science Department, Cornell University

Representation Learning Robot Imitation Learning Invariant Representation Learning Self-Supervised Visual Representation Learning

Feb 26, 2026cs.CV

SO3UFormer: Learning Intrinsic Spherical Features for Rotation-Robust Panoramic Dense Prediction

Panoramic dense-prediction models, spanning semantic segmentation and depth estimation, are typically trained under a strict gravity-aligned assumption. Real-world captures, however, routinely violate it: handheld devices jitter and aerial platforms change attitude, so the camera is rarely upright. Under such 3D reorientation, standard spherical Transformers overfit global latitude cues and collapse. We introduce SO3UFormer, an architecture that learns intrinsic spherical features largely decoupled from the underlying coordinate frame, through three geometric components: (1) removing absolute latitude encoding, which breaks the dependence on the gravity axis; (2) quadrature-consistent spherical attention, which corrects for non-uniform sampling density; and (3) a gauge-aware relative positional bias built from local tangent-plane angles rather than global axes. A logit-space \emph{SO(3)}-consistency regularizer, used only during training, further suppresses residual discretization effects. To benchmark robustness, we introduce Pose35, a variant of Stanford2D3D perturbed by random rotations within

\pm 35^\circ

, and evaluate under a full, arbitrary \emph{SO(3)} stress test. There, the baseline SphereUFormer collapses from 67.53 \emph{mIoU} on Pose35 to 25.26 under the full \emph{SO(3)} test, whereas SO3UFormer reaches 72.03 on Pose35 and retains 70.67 under the same test. Similarly, on a second real-world dataset (Matterport3D) for segmentation and on panoramic depth estimation, SO3UFormer remains essentially rotation-invariant while the gravity-anchored baseline again loses most of its accuracy. Code and models are available at https://github.com/zhuqinfeng1999/SO3UFormer.

Qinfeng Zhu, Yunxi Jiang, Lei Fan

University of Liverpool, United Kingdom. · CNRS, France. · Xi’an Jiaotong-Liverpool University, China.

Depth Estimation Vision Transformer Invariant Representation Learning Semantic Segmentation

Feb 26, 2026cs.LG

Transformers converge to invariant algorithmic cores

Training selects for behavior, not circuitry: many weight configurations can implement the same function. Studying any single trained neural network thus risks describing accidents of one training run rather than the computation itself. This work shifts focus from what transformers happen to do to what they must do by extracting algorithmic cores, compact subspaces that are necessary and sufficient for a task and that recur across independently trained models. Here, Algorithmic Core Extraction (ACE) is introduced to isolate these subspaces, causally validate them, and recover the algorithms they implement across settings ranging from synthetic tasks to large-scale pretrained models. Markov-chain transformers embed three-dimensional cores in nearly orthogonal subspaces yet recover identical transition spectra. Modular-addition transformers form compact cyclic cores at grokking that later inflate under continued regularization, redundantly distributing the same computation across many functionally equivalent modes. This functional redundancy is found to accelerate the transition from memorization to generalization, yielding an inverse scaling law for grokking time. In six language models spanning more than two orders of magnitude in scale (GPT-2 Small/Medium/Large, LLaMA-3.1, Gemma-2, and Qwen2.5), subject-verb agreement is governed by a single, steerable axis that aligns across architectures. Flipping this axis inverts grammatical number throughout open-ended generation. Together these results suggest that beneath the apparent complexity of trained transformers lies a simpler, shared computational structure, and that targeting invariants rather than parameterizations may offer a more tractable path to mechanistic understanding and control. Code: https://github.com/joshseth/cores

Joshua S. Schiffman

New York Genome Center, New York, NY, USA.

Transformer Interpretability Invariant Representation Learning Grokking

Feb 2, 2026cs.LG

Self-Supervised Learning from Structural Invariance

Joint-embedding self-supervised learning (SSL), the key paradigm for unsupervised representation learning from visual data, learns from invariances between semantically-related data pairs. We study the one-to-many mapping problem in SSL, where each datum may be mapped to multiple valid targets. This arises when data pairs come from naturally occurring generative processes, e.g., successive video frames. We show that existing methods struggle to flexibly capture this conditional uncertainty. As a remedy, we introduce a latent variable to account for this uncertainty and derive a variational lower bound on the mutual information between paired embeddings. Our derivation yields a simple regularization term for standard SSL objectives. The resulting method, which we call AdaSSL, applies to both contrastive and distillation-based SSL objectives, and we empirically show its versatility in causal representation learning, fine-grained image understanding, and world modeling on videos.

Yipeng Zhang, Hafez Ghaemi, Jungyoon Lee +3

Mila - Qu´ebec AI Institute · Universit´e de Montr´eal · CHU Sainte-Justine +2

Unsupervised Learning Self-Supervised Learning Invariant Representation Learning

Feb 2, 2026cs.CV

Rotation-free Online Handwritten Character Recognition Using Linear Recurrent Units

Online handwritten character recognition leverages stroke order and dynamic features, which generally provide higher accuracy and robustness compared with offline recognition. However, in practical applications, rotational deformations can disrupt the spatial layout of strokes, substantially reducing recognition accuracy. Extracting rotation-invariant features therefore remains a challenging open problem. In this work, we employ the Sliding Window Path Signature (SW-PS) to capture local structural features of characters, and introduce the lightweight Linear Recurrent Units (LRU) as the classifier. The LRU combine the fast incremental processing capability of recurrent neural networks (RNN) with the efficient parallel training of state space models (SSM), while reliably modelling dynamic stroke characteristics. We conducted recognition experiments with random rotation angle up to

\pm 180^{\circ}

on three subsets of the CASIA-OLHWDB1.1 dataset: digits, English upper letters, and Chinese radicals. The accuracies achieved after ensemble learning were

99.62\%

,

96.67\%

, and

94.33\%

, respectively. Experimental results demonstrate that the proposed SW-PS+LRU framework consistently surpasses competing models in both convergence speed and test accuracy.

Zhe Ling, Sicheng Yu, Danyu Yang

College of Mathematics and Statistics, Chongqing University, China

Online Handwriting Recognition Invariant Representation Learning Optical Character Recognition

Jan 21, 2026cond-mat.dis-nn

Learning and extrapolating scale-invariant processes

Machine Learning (ML) has deeply changed some fields recently, like Language and Vision and we may expect it to be relevant also to the analysis of of complex systems. Here we want to tackle the question of how and to which extent can one regress scale-free processes, i.e. processes displaying power law behavior, like earthquakes or avalanches? We are interested in predicting the large ones, i.e. rare events in the training set which therefore require extrapolation capabilities of the model. For this we consider two paradigmatic problems that are statistically self-similar. The first one is a 2-dimensional fractional Gaussian field obeying linear dynamics, self-similar by construction and amenable to exact analysis. The second one is the Abelian sandpile model, exhibiting self-organized criticality. The emerging paradigm of Geometric Deep Learning shows that including known symmetries into the model's architecture is key to success. Here one may hope to extrapolate only by leveraging scale invariance. This is however a peculiar symmetry, as it involves possibly non-trivial coarse-graining operations and anomalous scaling. We perform experiments on various existing architectures like U-net, Riesz network (scale invariant by construction), or our own proposals: a wavelet-decomposition based Graph Neural Network (with discrete scale symmetry), a Fourier embedding layer and a Fourier-Mellin Neural Operator. Based on these experiments and a complete characterization of the linear case, we identify the main issues relative to spectral biases and coarse-grained representations, and discuss how to alleviate them with the relevant inductive biases.

Anaclara Alvez-Canepa, Cyril Furtlehner, François P. Landes

Laboratoire Interdisciplinaire des Sciences du Numérique (LISN) · Université Paris-Saclay and · INRIA-Saclay

Invariant Representation Learning Neural Operators

Jan 17, 2026cs.CV

Learning Language-Driven Sequence-Level Modal-Invariant Representations for Video-Based Visible-Infrared Person Re-Identification

The core of video-based visible-infrared person re-identification (VVI-ReID) lies in learning sequence-level modal-invariant representations across different modalities. Recent research tends to use modality-shared language prompts generated by CLIP to guide the learning of modal-invariant representations. Despite achieving optimal performance, such methods still face limitations in efficient spatial-temporal modeling, sufficient cross-modal interaction, and explicit modality-level loss guidance. To address these issues, we propose the language-driven sequence-level modal-invariant representation learning (LSMRL) method, which includes spatial-temporal feature learning (STFL) module, semantic diffusion (SD) module and cross-modal interaction (CMI) module. To enable parameter- and computation-efficient spatial-temporal modeling, the STFL module is built upon CLIP with minimal modifications. To achieve sufficient cross-modal interaction and enhance the learning of modal-invariant features, the SD module is proposed to diffuse modality-shared language prompts into visible and infrared features to establish preliminary modal consistency. The CMI module is further developed to leverage bidirectional cross-modal self-attention to eliminate residual modality gaps and refine modal-invariant representations. To explicitly enhance the learning of modal-invariant representations, two modality-level losses are introduced to improve the features' discriminative ability and their generalization to unseen categories. Extensive experiments on large-scale VVI-ReID datasets demonstrate the superiority of LSMRL over AOTA methods.

Xiaomei Yang, Antai Liu, Xizhan Gao +3

Shandong Key Laboratory of Ubiquitous Intelligent Computing, School of Information Science and Engineering, University of Jinan, Jinan 250022, China · College of Information Science and Technology & Artificial Intelligence, Nanjing Forestry University, Nanjing 210037, China · Department of Informatics, Modeling, Electronics, and Systems, University of Calabria, Via P. Bucci, cubo 41C, 87036, Rende (CS), Italy

Invariant Representation Learning Person Re-Identification

Jan 6, 2026cs.CV

Higher order PCA-like rotation-invariant features for detailed shape descriptors modulo rotation

PCA can be used for rotation invariant features, describing a shape with its

p_{ab}=E[(x_i-E[x_a])(x_b-E[x_b])]

covariance matrix approximating shape by ellipsoid, allowing for rotation invariants like its traces of powers. However, real shapes are usually much more complicated, hence there is proposed its extension to e.g.

p_{abc}=E[(x_a-E[x_a])(x_b-E[x_b])(x_c-E[x_c])]

order-3 or higher tensors describing central moments, or polynomial times Gaussian allowing decodable shape descriptors of arbitrarily high accuracy, and their analogous rotation invariants. Its practical applications could be rotation-invariant features to include shape modulo rotation e.g. for molecular shape descriptors, or for up to rotation object recognition in 2D images/3D scans maybe also for 3D scene understanding, or shape similarity metric allowing inexpensive comparison of objects modulo rotation avoiding costly optimization over rotations.

Jarek Duda

Jagiellonian University, Krakow, Poland

Principal Component Analysis Invariant Representation Learning

Dec 16, 2025cs.LG

Implicit Bias and Invariance: How Hopfield Networks Efficiently Learn Graph Orbits

Many learning problems are organized by group symmetries. While invariance is often imposed through architectures or group averaging, we ask when it can emerge from training on a finite random subset of an orbit. We study this question in classical Hopfield networks, where strict memorization can be expressed as a linear margin problem. Reparameterizing minimization of energy flow (MEF) as an exponential loss connects gradient descent to the corresponding minimum-norm hard-margin memorizer. Our main result shows that, for independent uniform samples from any finite permutation orbit, the exact sample hard-margin support vector machine (HSVM) concentrates exponentially around the invariant full-orbit HSVM. Consequently, an orbit-size-independent polynomial number of samples suffices both for approximate parameter invariance and for simultaneous memorization of every orbit element; directional convergence transfers this conclusion asymptotically to MEF gradient descent. For graph-isomorphism orbits, we characterize the invariant parameters as a three-dimensional subspace and show that every such orbit is memorizable. For cliques of fixed linear density, additional symmetry sharpens the uniform memorization bound to

O(v^4\log(1/δ))

, exponentially smaller than the orbit size. Together with experiments across several learning rules, these results give a finite-sample account of how optimization bias can recover symmetry from partial group-structured data.

Michael Murray, Tenzin Chan, Kedar Karhadker +1

Mathematical Sciences, University of Bath · Department of Mathematics, UCLA · Algebraic

Sample Complexity Neural Network Memorization Hopfield Networks Implicit Bias Permutation-Invariant Learning Invariant Representation Learning

Sep 28, 2025cs.LG

GraphIFE: Rethinking Graph Imbalance Node Classification via Invariant Learning

The class imbalance problem refers to the disproportionate distribution of samples across different classes within a dataset, where the minority classes are significantly underrepresented. This issue is also prevalent in graph-structured data. Most graph neural networks (GNNs) implicitly assume a balanced class distribution and therefore often fail to account for the challenges introduced by class imbalance, which can lead to biased learning and degraded performance on minority classes. We identify a quality inconsistency problem in synthesized nodes, which leads to suboptimal performance under graph imbalance conditions. To mitigate this issue, we propose GraphIFE (Graph Invariant Feature Extraction), a novel framework designed to mitigate quality inconsistency in synthesized nodes. Our approach incorporates two key concepts from graph invariant learning and introduces strategies to strengthen the embedding space representation, thereby enhancing the model's ability to identify invariant features. Extensive experiments demonstrate the framework's efficiency and robust generalization, as GraphIFE consistently outperforms various baselines across multiple datasets. The code is publicly available at https://github.com/flzeng1/GraphIFE.

Fanlong Zeng, Wensheng Gan, Kangjie Chen +1

School of Intelligent Systems Science and Engineering, Jinan University, Zhuhai 519070, China · School of Cybersecurity, Tianjin University, Tianjin 300354, China · University of Illinois Chicago, Chicago, USA

Class-Imbalanced Learning Graph Neural Networks Data Augmentation Invariant Representation Learning Node Classification

Feb 26, 2025cs.LG

Invariance Pair Guidance: Robustness to Spurious Correlations via Corrective Gradients

Machine learning models are inherently bound to the distribution of the training data, often exploiting non-causal shortcuts. As a result, achieving robustness to spurious correlations remains a challenge. While existing approaches rely on data manipulation or re-weighting strategies to achieve robustness, they typically require dense group labels, multiple training domains, or specialized pre-processing. We propose Invariance Pair Guidance (IPG), a method to mitigate reliance on spurious correlations using a sparse set of counterfactual pairs. Unlike other methods demanding extensive supervision, IPG utilizes a novel dual-update mechanism to dynamically correct the optimization trajectory. We generate input pairs that isolate the spurious attribute to define the invariance, a characteristic that should not affect the outcome of the model. Based on these pairs, we define a corrective gradient that complements the traditional gradient descent approach. The correction adapts via a predefined invariance condition. Experiments on ColoredMNIST, Waterbirds-100, and CelebA datasets demonstrate the effectiveness of our approach and its robustness to group shifts, supported by a theoretical convergence analysis. IPG offers a data-efficient and theoretically grounded path to robustness.

Martin Surner, Abdelmajid Khelil, Ludwig Bothmann

Institute for Data and Process Science, Landshut University of Applied Sciences, Landshut, Germany. · Computer Science Department, Landshut University of Applied Sciences, Landshut, Germany. · Department of Statistics, LMU Munich, Munich, Germany. +1

Spurious Correlation Robustness Invariant Representation Learning

Feb 20, 2025stat.ML

Bringing Generative Learning to Representation Learning: Self-Supervised Transfer Learning as Distribution Matching

Most self-supervised learning objectives defend against collapse but leave the target representation law unspecified. We formulate representation learning as Distribution Matching (DM), learning an augmentation-invariant encoder whose induced law matches an explicit geometric reference. The reference law specifies what the learned representation distribution should look like, whereas a separately chosen discrepancy determines how deviations from this target are measured; here we use Mallows distance. The DM framework reveals a directional inverse: generative learning maps a tractable reference to data, whereas representation learning maps data to a designed reference law. We connect the population objective to class-centre separation and classification error and prove a non-asymptotic neural-sieve guarantee. Simulations and image benchmarks show manifold rectification, fine-grained structure and transfer across label spaces.

Yuling Jiao, Wensen Ma, Defeng Sun +2

School of Artificial Intelligence, Wuhan University, 430072, Wuhan, China · School of Mathematics and Statistics, Wuhan University, Wuhan, 430072, China · Department of Applied Mathematics, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong SAR, China +2

Visual Representation Learning Distribution Matching Transfer Learning Invariant Representation Learning Self-Supervised Visual Representation Learning