Invariant Representation Learning

Overview

Invariant representation learning studies how to encode data so that changes such as viewpoint, sensor, speaker, environment, or input format do not obscure task-relevant information. Approaches include symmetry-aware architectures, contrastive and predictive self-supervision, and methods that disentangle stable features from domain-specific variation or align representations across environments. These ideas are used in vision, graphs, speech, biomedical signals, robotics, and scientific data, often evaluated on shifted domains, unseen transformations, or cross-subject and cross-device benchmarks. A central tension is that removing variation can improve robustness but also discard useful signal; invariance alone does not guarantee reliable transfer, fairness, or generalization when the target itself changes across environments.

Latest papers

242 papers13 papers in the last four weeks, down 63% on the four weeks before. 0.1% of all new papers.

May 29, 2024

Invariant Graph Representations for Continuous-Time Dynamic Graphs Under Distribution Shifts

Continuous-Time Dynamic Graphs (CTDGs) enable fine-grained modeling of evolving relational systems. However, most existing CTDG representation learning methods are tailored to in-distribution settings and exhibit limited robustness under out-of-distribution (OOD) shifts. Although recent causal approaches learn invariant representations via interventions, they are primarily designed for static or discrete-time graphs and become computationally prohibitive for CTDGs due to the combinatorial explosion of structural and temporal variations. To address these challenges, we propose CIR, a framework grounded in a novel structural causal model termed the ICCM. To avoid exhaustive interventions, we leverage the Normalized Weighted Geometric Mean (NWGM) to efficiently approximate interventional predictions. We further instantiate ICCM within a practical deep learning architecture that jointly captures invariant structural and temporal patterns through dedicated subgraph extractors, and maintains an environment memory bank to model distributional shifts across evolving contexts. Extensive experiments demonstrate that CIR consistently outperforms existing methods under diverse OOD scenarios.
Oct 2, 2022

Tensor-reduced atomic density representations

Density based representations of atomic environments that are invariant under Euclidean symmetries have become a widely used tool in the machine learning of interatomic potentials, broader data-driven atomistic modelling and the visualisation and analysis of materials datasets.The standard mechanism used to incorporate chemical element information is to create separate densities for each element and form tensor products between them. This leads to a steep scaling in the size of the representation as the number of elements increases. Graph neural networks, which do not explicitly use density representations, escape this scaling by mapping the chemical element information into a fixed dimensional space in a learnable way. We recast this approach as tensor factorisation by exploiting the tensor structure of standard neighbour density based descriptors. In doing so, we form compact tensor-reduced representations whose size does not depend on the number of chemical elements, but remain systematically convergeable and are therefore applicable to a wide range of data analysis and regression tasks.