Equivariant Representation Learning

Follow this topic Ask about this topic

Momentum

5 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 30

Oct 7, 2026cs.LG

How Do Transformers Learn to Represent Symmetries?

Training Transformer-based architectures with finite data augmentation has become an increasingly popular approach in geometric machine learning. Despite its empirical success, the interplay between the Transformer architecture, invariance to different symmetries, and augmentation budgets remains underexplored. In this paper, we study the ability of a vanilla Transformer to learn various symmetries through finite data augmentation for point cloud datasets. We identify an ordering of increasing learnability across the following symmetry groups: (i) non-angle-preserving symmetries, (ii) angle-preserving symmetries, and (iii) base angle-preserving subgroups, such as translation, rotation, and scale. For the base angle-preserving groups, we further investigate the Transformer's extrapolation behavior and conduct a structural analysis of the trained models, allowing us to identify interpretable mechanisms that induce invariance. Finally, we extend our analysis to equivariant functions and show that the detected mechanisms for approximate invariance can also provide a key building block for learned equivariance. Our project page is available at https://transformers-learn-symmetries.github.io/

Eduardo Santos-Escriche, Valerie Engelmayer, Ya-Wei Eileen Lin +1

Technical University of Munich Munich Center for Machine Learning · Massachusetts Institute of Technology

Invariant Representation Learning Transformer Equivariant Representation Learning Data Augmentation

Oct 7, 2026physics.chem-ph

PEACE: Covariant learning of nonadiabatic manifolds with parity-resolved Hamiltonians

Nonadiabatic molecular dynamics provides mechanistic insight into light-driven processes and informs the design of molecules and materials for solar energy conversion, photocatalysis and photo switching. Accurately describing these processes requires a representation that respects electronic symmetry and consistently relates energies to interstate couplings. Here we introduce PEACE, which combines a parity-equivariant latent Hamiltonian with a learned electronic connection. Controlled ablations reveal the complementary roles of symmetry-allowed state mixing and electronic-frame variation in reproducing crossing structures and relaxation dynamics. PEACE closely reproduces excited-state population dynamics from first-principles simulations, while its extension to spin-orbit coupling enables simulations of intersystem crossing. These results demonstrate that a more complete incorporation of the underlying physics into learned electronic representations leads to more accurate predictions of nonadiabatic dynamics.

Rongzhi Gao, Shuguang Chen, Yang Zhou +3

Department of Chemistry, The University of Hong Kong, Pok Fu Lam, Hong Kong SAR, China · Hong Kong Quantum AI Lab, Pak Shek Kok, Hong Kong SAR, China · MattVerse Limited, Pak Shek Kok, Hong Kong SAR, China +1

Molecular Dynamics Simulation Quantum Chemistry Equivariant Representation Learning

Oct 6, 2026cs.LG

Symmetry-Aware Feature Learning: A Polynomial Separation for Multi-Index Models

We establish a polynomial sample complexity separation between symmetry-aware and symmetry-agnostic feature learning. We study growing-rank multi-index models with high-dimensional Gaussian covariates in

\mathbb{R}^d

and

r=Θ(d^δ)

teacher directions forming a cyclic symmetry orbit, where

0<δ<1/2

. We compare three ways of exploiting this structure: architectural weight sharing, data augmentation over the full symmetry group, and learning without access to the symmetry. In particular, we analyze a symmetry-tied convolutional network, an untied network, and the same untied network trained with full-group data augmentation, using spherical online SGD with correlation loss. For a class of polynomial links with information exponent

p\ge3

, we prove matching sample complexity bounds up to logarithmic factors: the tied and augmented learners achieve weak directional recovery in

\widetildeΘ(d^{p-1})

samples, whereas the symmetry-agnostic learner requires

\widetildeΘ(rd^{p-1})

. For the pure quadratic Hermite link, the same separation holds for weak recovery of the teacher subspace, with sample complexities

\widetildeΘ(d)

and

\widetildeΘ(rd)

, respectively. Thus, full-group data augmentation matches the sample efficiency of architectural weight sharing, and both provide a polynomial advantage over training without symmetry. For

p\ge3

, the proof reveals a two-stage mechanism: fluctuations at initialization select one direction in the teacher orbit, after which localized growth amplifies its overlap to the weak recovery scale while competing overlaps remain near their initialization scale.

Jivan Waber, Vanessa Piccolo, Yatin Dandi +1

Information, Learning and Physics Laboratory École Polytechnique Fédérale de Lausanne (EPFL) · Statistical Physics of Computation Laboratory École Polytechnique Fédérale de Lausanne (EPFL)

Representation Learning Data Augmentation Equivariant Representation Learning Statistical Learning Theory Multi-Index Models

Oct 6, 2026cs.CV

REViT-v2: Hierarchical Windowed Roto-reflection Equivariant ViT for Equivariant Feature Extraction

We propose a scalable roto-reflection-group-equivariant vision transformer based on windowed group-convolutional self-attention and a hierarchical feature architecture. We demonstrate that our approach can be scaled to group-equivariant vision transformers (ViTs) with millions of parameters and large datasets with practically sized images, i.e., ImageNet. The code and pretrained weights for the proposed Hierarchical Windowed Roto-reflection Equivariant ViTs (REViT-v2) are available at https://github.com/kc-ml2/revit.

Sheir A. Zaheer, Jihwan Moon, Chan Y. Park

KC Machine Learning Lab, Rep. of Korea · NFOCZ Inc and Seoul National University, Rep. of Korea

Vision Transformer Equivariant Neural Networks Equivariant Representation Learning

Oct 1, 2026cs.SD

From Isolated Feature to Orbits: Discovering Music Concepts via Multi-SAE Alignment

How can we understand what a music foundation model has learned \textit{internally}? Most interpretability approaches, such as probing and Sparse Autoencoders (SAEs), focus on identifying individual features with minimal structural assumptions. We argue that many concepts are better understood as \textit{structured relations} rather than isolated features. This is especially prominent in music, where tonal structures are organized in the space of pitch and time. For example, concepts such as chords or keys are naturally expressed as structured sets (e.g., the 12 transpositions of a chord or the diatonic system within a key), rather than isolated features. In this study, \textbf{we shift from feature identification to structure-based analysis}, asking whether the learned inner representations of music foundation model emerge as organized structures over features. To this end, we introduce a framework that uses pitch transposition as an inductive bias to induce ordered orbits via multi-view SAE alignment. Concretely, we generate pitch-shifted input pairs and align their SAE representations to discover structured groups of pitch-related features. Experimental results show that this approach recovers orbit structures corresponding to chords, keys, and melodic patterns across two state-of-the-art music foundation models, while requiring only minimal grounding (e.g., a few anchor examples) to interpret entire concept families.

Liwei Lin, Gus Xia

Computer Science Department, NYU Shanghai · Machine Learning Department, MBZUAI

Representation Learning Sparse Autoencoders Mechanistic Interpretability Equivariant Representation Learning

Sep 30, 2026cs.LG

Reformulation-Contrastive Learning for Mixed Integer Programs

Mixed-integer linear programs (MILP) model many real-world decision problems, motivating machine-learning methods that exploit recurring structure to accelerate MILP solving. MILPs can admit many equivalent formulations: integrality-preserving changes of variables and the addition of redundant constraints can alter their formulations while preserving the optimization problem. We leverage these reformulations as a source of self-supervision for learning general-purpose representations of MILP variables and constraints. We characterize the affine reformulations that are valid for every input instance, and distinguish re-descriptions, which leave variables unchanged, from substitutions, which transform them predictably. Building on equivariant self-supervised learning, we introduce ReMILP (reformulation-contrastive MILP representation learning), which jointly trains a graph neural network and a hypernetwork to predict how variable embeddings transform under changes of variables. Without solver-derived labels, ReMILP learns representations that exhibit the intended invariance and equivariance on unseen problem classes. Across binary solution, constraint activity and integrality gap prediction, these representations carry task-relevant information when frozen and provide a useful initialization for fine-tuning.

Ousema Bouaneni, Mathis Le Bail, Clément Elliker +2

LIX (École Polytechnique, IP Paris, CNRS) · AMIAD (Agence Ministérielle pour l’IA de Défense)

Contrastive Learning Representation Learning Graph Representation Learning Self-Supervised Learning Mixed-Integer Linear Programming Equivariant Representation Learning

Sep 22, 2026math.RT

Graded Representation Theory of Equivariant Neural Networks

Nonlinear activations can create equivariant interactions between irreducible representations that linear maps cannot. We use the Gaussian degree decomposition to extend ordinary polynomial degree to such nonlinear maps, and prove that for a fixed coordinatewise equivariant layer each degree factors into a polynomial determined by the linear maps and a scalar determined by the activation. This separates three distinct obstructions, coming from symmetry, coordinates, and activation.

Mani Shayestehfar

Equivariant Neural Networks Equivariant Representation Learning Neural Network Expressivity

Sep 20, 2026cs.RO

BiRoAD: Learning Shared and Role-Adaptive Representations for Bimanual Manipulation

Bimanual manipulation requires policies that coordinate two arms while adapting their functional roles to scene geometry, object configuration, and task context. Learning such scene-conditioned role adaptation remains challenging, as demonstrations may contain uneven role distributions that limit generalization to underrepresented arm--role configurations. In addition, many bimanual policies predict actions in fixed left- and right-arm action spaces. While this provides a natural parameterization for robot control, it does not explicitly specify how behaviors should transform when functional roles are exchanged across arms. Across different scene initializations, the two arms may follow a similar coordination pattern, but the role-specific behavior assigned to each arm should change with the scene. Therefore, we propose BiRoAD, a Bimanual Role-Adaptive Decomposition framework for learning shared and role-adaptive representations in bimanual policies. Given bimanual trajectory or action-token features, BiRoAD decomposes these features into swap--symmetric and swap--antisymmetric components: the former captures coordination structure invariant to arm exchange, and the latter captures role-specific distinctions that vary consistently with functional role assignment. The two components are then recomposed as residual updates to the original paired arm representations, allowing BiRoAD to serve as a modular feature transformation without changing the policy inputs, imitation-learning objective, or requiring manually defined role labels. Across multiple bimanual manipulation tasks with balanced and imbalanced role distributions, BiRoAD improves robustness across role configurations over corresponding base policies, with notable gains on underrepresented role configurations.

Yan Shen, Yuchen Liu, Feng Jiang +5

CFCS, School of Computer Science, Peking University

Representation Learning Bimanual Robotic Manipulation Generalization in Robotic Manipulation Robot Imitation Learning Motion Representation Learning Equivariant Representation Learning

Sep 8, 2026cs.LG

Equivariance Breaks the Learning Rate

Equivariant networks are commonly trained with Adam, yet recent work reports that matrix structured optimizers such as Muon can perform better, with the reasons for these gains only partly understood. We identify one source of this difference inside equivariant layers. An equivariant layer learns one channel mixing matrix

W_l

per degree

l

, which we call an irrep block, and shares it across the

2l+1

components, giving the expanded map

W_l \otimes I_{2l+1}

. This sharing sums gradient contributions across components and can produce different update scales under SGD. Adam's entrywise normalization reduces sensitivity to gradient scale, but neither optimizer directly controls the effective step size of each block. A single learning rate can therefore produce different effective step sizes across blocks. Muon instead controls the effective step size by approximately equalizing the singular values of each momentum matrix. We normalize each irrep block update by a single scalar, preserving its singular value ratios while letting the learning rate control its size. We implement this with spectral normalization or a simpler root-mean-square normalization. We evaluate spectral normalization in a controlled

\mathrm{SO}(3)

-equivariant model with a matched non-equivariant model. In this setting, the step size mismatch grows with width in the equivariant model but not in the non-equivariant model. We evaluate both variants across molecular force prediction on the rMD17 and MD22 datasets, QM9 molecular property prediction, and charged particle dynamics. Across these applications, block normalization generally improves Adam and closes part of its gap to Muon. These results highlight an overlooked interaction between equivariant architectures and their optimizers. Studying and designing the two together may help explain and address training difficulties often attributed to equivariance itself.

Andrei Manolache, Mathias Niepert

University of Stuttgart, Germany · Bitdefender, Romania

Equivariant Neural Networks Equivariant Representation Learning Adaptive Gradient Methods

Sep 1, 2026cs.CV

Pix2Rep-v2: Data-Efficient Representation Learning for Dense Medical Imaging Applications

Dense self-supervised learning (SSL) is a powerful paradigm for learning without annotations the local descriptors required to solve dense medical imaging tasks. We present Pix2Rep-v2, a framework for SSL of pixel- and voxel-level representations suitable for few-shot downstream applications. Pix2Rep-v2 addresses the main challenges of dense SSL by leveraging a redundancy reduction objective at the pixel-level with a principle of equivariance of dense representations, that scales efficiently to 3D or wide field-of-view applications. We evaluate our method on four datasets, across multiple tasks, multiple modalities and anatomical structures using multiple backbones in 2D and 3D, and under various data regimes. As an alternative to linear probing or full fine-tuning on the downstream task, we also propose an in-context variant, without downstream training, based on a dense prototype approach. Pix2Rep-v2 shows substantially higher data-efficiency in few-shot scenarios compared to fully supervised baselines, and is competitive with the state-of-the-art e.g., +9.3 Dice points in one-shot segmentation on the M&Ms-2 dataset. Our code and pre-trained models are publicly available at https://github.com/BioMedTP/pix2rep-v2.

S. Sifaoui, E. Angelini, S. Toupin +2

LTCI, Télécom Paris, Institut Polytechnique de Paris, Palaiseau, France · MIRACL.ai Laboratory, Hôpital Universitaire Lariboisière (AP-HP), Paris, France · Université Paris Cité, Inserm MASCOT, Paris, France

Representation Learning Visual Representation Learning Equivariant Representation Learning Few-Shot Medical Image Segmentation Medical Image Representation Learning Self-Supervised Visual Representation Learning

Aug 31, 2026cs.LG

Rotational Equivariance in Machine Learning: A Comprehensive Tutorial

Rotational symmetry is one of the most important structural principles in machine learning on 3D data. In applications ranging from physics and materials science to 3D computer vision, predictions should not depend on an arbitrary choice of coordinate frame. Rotational equivariance captures this requirement mathematically by enforcing that a rotation of the input induces a corresponding transformation of the model output. This tutorial provides a comprehensive introduction to rotational equivariance, starting from the physical and geometric intuition behind coordinate independence and building up the necessary machinery from geometric deep learning, group theory, and representation theory. We introduce message passing on Euclidean graphs, group actions and representations, spherical harmonics, Wigner matrices, tensor products, and Clebsch-Gordan decomposition, and explain how these ingredients give rise to modern equivariant architectures. We then survey the principal strategies for incorporating rotational equivariance in deep learning, including group convolutions, internal tensorial representations, and canonicalization-based methods, and discuss their practical strengths and limitations. The tutorial aims to lower the barrier to the subject by connecting the underlying mathematics to practical model design, by unifying ideas that are often expressed in different formal languages, and by helping practitioners choose among competing approaches through a clear discussion of their trade-offs.

Peter Lippmann, Fred A. Hamprecht

Interdisciplinary Center for Scientific Computing (IWR), Heidelberg University, 69120 Heidelberg, Germany

Equivariant Neural Networks Geometric Representation Learning 3D Representation Learning Equivariant Representation Learning

Aug 31, 2026cs.LG

No Equivariant Architecture Covers All Equivariant Attention

We give a complete characterization of equivariant multi-head self-attention (MHSA): if an MHSA layer is equivariant to a symmetry group

G

, then

G

can only act by permuting head-clusters, with QK and OV matrices satisfying an equivariance constraint tied to the group action. As a consequence, we prove that any fixed MHSA architecture that achieves exact equivariance by polynomially parameterizing unconstrained MHSA parameters inevitably leads to expressivity loss within the class of equivariant maps: the equivariance locus of unconstrained MHSA forms a union of extremely many Zariski-irreducible components in a reduced parameter space, and any single architecture covers at most one. For

G=D_4

acting on

C

copies of the regular representation as the token feature space, we show that there are

Ω(C^{64})

components for eight attention heads.

Tīkun Ông

Attention Head Analysis Equivariant Neural Networks Equivariant Representation Learning Multi-Head Attention

Aug 28, 2026cs.LG

Equivariant Sheaf Neural Networks: Learning Geometric Transport on Graphs

Equivariant graph neural networks provide a principled way to model geometric systems, but efficient first-order architectures remain limited in how vector information can be transformed as it moves across a graph. We introduce \textsc{ESNN}, an Equivariant Sheaf Neural Network that enriches this interaction by learning directed, matrix-valued transport between neighboring vector features while preserving exact Euclidean equivariance. Rather than increasing the order of the representation, ESNN keeps scalar and vector features first-order and places the additional geometric flexibility in the edge transport itself. We characterize this transport theoretically, showing that when relative displacement is the only covariant geometric input, every linear

O(n)

-equivariant map decomposes into independent radial and tangential components, while learned covariant features enable richer feature-conditioned transformations. We also introduce controlled symmetry relaxation for systems with a preferred ambient direction, which may be prescribed or inferred from data while recovering full

E(n)

-equivariance when the directional pathway is inactive. Across particle dynamics, mesh-based simulation, point-cloud classification, and molecular property prediction, ESNN improves dynamics prediction, recovers the gravity axis when symmetry is broken, yields substantial gains on selected mesh tasks and long-horizon rollouts, and remains robust to unseen rotations. These results show that learning how geometric information is transported across edges offers a complementary route to expressive equivariant message passing without requiring higher-order representations.

Alessio Borgi, Mario Severino, Fabrizio Silvestri +1

Department of Computer Science and Technology, University of Cambridge · Department of Computer, Control and Management Engineering, Sapienza University of Rome · Department of Information Engineering, University of Padua

Equivariant Neural Networks Sheaf Neural Networks Equivariant GNNs Equivariant Representation Learning

Aug 12, 2026cs.LG

Reducing Symmetry Increase in Equivariant Neural Networks

Equivariant Neural Networks (ENNs) have empowered numerous applications in scientific fields. Despite their remarkable capacity for representing geometric structures, ENNs suffer from degraded expressivity when processing symmetric inputs: the output representations are invariant to transformations that extend beyond the input's symmetries. The mathematical essence of this phenomenon is that a symmetric input, after being processed by an equivariant map, experiences an increase in symmetry. While prior research has documented symmetry increase in specific cases, a rigorous understanding of its underlying causes and general reduction strategies remains lacking. In this paper, we provide a detailed and in-depth characterization of symmetry increase together with a principled framework for its reduction: (i) For any given feature space and input symmetry group, we prove that the increased symmetry admits an infimum determined by the structure of the feature space; (ii) Building on this foundation, we develop a computable algorithm to derive this infimum, and propose practical guidelines for feature design to prevent harmful symmetry increases. (iii) Under standard regularity assumptions, we demonstrate that for most equivariant maps, our guidelines effectively reduce symmetry increase. To complement our theoretical findings, we provide visualizations and experiments on both synthetic datasets and the real-world QM9 dataset. The results validate our theoretical predictions.

Ning Lin, Jiacheng Cen, Anyi Li +2

Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China

Equivariant Neural Networks Equivariant Representation Learning Neural Network Expressivity

Aug 4, 2026cs.SD

Equivariant Music Transformer

Humans recognize a musical passage even when it is shifted in time or transposed in pitch, indicating a notion of equivariance in the representation space. Our analysis, however, shows that standard music transformers map such time-shifted or pitch-transposed inputs onto uncorrelated representations: these models become progressively less equivariant as they scale in size or train longer. This suggests that in standard music transformers, additional model capacity is allocated to memorizing absolute patterns rather than capturing shared musical structures. In this paper, we propose the Equivariant Music Transformer (EMT), which enforces equivariance through self-distillation by jointly optimizing a next-token-prediction and an auxiliary equivariance regularization loss. We find that the additional equivariance loss acts as a beneficial regularizer, simultaneously improving next-token prediction and producing equivariant latent representations. Through both objective and subjective evaluations, EMT demonstrates superior equivariance and generative capability compared to data augmentation, feature engineering, and state-of-the-art (SOTA) baselines. More broadly, our findings reveal that standard language modeling methods alone do not capture music's translational symmetries, and dedicated inductive biases are required to produce better music representations. The code, weights and demos are available online.

Zixun Guo, Simon Dixon

Centre for Digital Music, Queen Mary University of London, United Kingdom

Representation Learning Transformer Symbolic Music Generation Equivariant Neural Networks Equivariant Representation Learning

Jul 28, 2026cs.LG

Generator-Aligned Representation Interfaces for Diagnostic Soft Equivariance

Exact-equivariant architectures typically encode prescribed group actions in specialized operators, which can complicate their reuse with generic backbones and across data modalities. We introduce the Generator-Aligned Representation Interface (GARI), a representation-level design principle that exposes selected transformation generators to a generic sequence backbone through aligned canonical and generator-induced views. We formalize the resulting behavior using a probe-specific soft-equivariance residual defined over declared data and transformation distributions. This framework distinguishes representation consistency from task robustness and exact equivariance, and localizes residual mismatch to interface construction, shared stream processing, and terminal fusion. We instantiate the interface as GARI-Net, which constructs generator-indexed streams, converts them into a common interaction frame, processes them with shared parameters, repairs ordering-induced context mismatch, enables cross-stream information exchange, and aggregates them using inter-stream discrepancy. Direct Equivariance Error (DEE) provides a frozen-checkpoint diagnostic of the prescribed representation relation under known token or voxel actions. Experiments on genomic sequences, images, and three-dimensional point clouds examine sequence reversal, planar rotations and reflections, and controlled axial transfer. Across these settings, the same interface principle supports task-relevant transformation consistency and generalization to declared held-out probes without requiring group-specific redesign of the sequence backbone. GARI therefore provides a portable diagnostic complement to hard-equivariant architectures: it makes generator structure accessible, learnable, and measurable, while finite-probe evidence remains distinct from certification of exact equivariance over a continuous group.

Weitao Li, Gong Cheng

School of Mathematical Sciences Tongji University Shanghai 200092, China · School of Mathematical Sciences Key Laboratory of Intelligent Computing and Applications (Ministry of Education) Tongji University Shanghai 200092, China

Equivariant Neural Networks Equivariant Representation Learning

Jul 21, 2026cs.LG

GEqTrain: A Configuration-Driven Framework for Retargeting Equivariant Graph Neural Networks Across 3D Scientific Tasks

Equivariant graph neural networks provide a powerful modeling language for three-dimensional scientific data, but their reuse is often limited by implementations tied to specific tasks, outputs, and training regimes. We present GEqTrain, a configuration-driven framework that separates dataset semantics, model composition, and training objectives. Raw data are mapped to typed node-, edge-, and graph-level fields, while model stacks, losses, and training workflows are assembled declaratively through Hydra configurations. A shared equivariant backbone and training infrastructure can therefore be retargeted to a new task primarily through configuration. We demonstrate this flexibility on three different problems handled within one software stack: coarse-grained-to-atomistic backmapping of biomolecular systems, prediction of NMR chemical shifts in molecular solids, and equivariant generative modeling. Our aim is not to surpass individually optimized task-specific systems, but to show that a shared representation and training infrastructure can achieve competitive accuracy across qualitatively different tasks at the cost of a configuration change. We further introduce GEqDiff, a generative extension based on equivariant flow matching. GEqDiff treats user-defined equivariant fields as first-class generation targets, jointly transporting Cartesian positions and non-scalar node fields spanning representations up to l=3 within a single equivariant flow. We validate this capability on a controlled synthetic benchmark inspired by protein secondary-structure motifs, showing that fields with heterogeneous transformation properties can be reconstructed jointly and with high fidelity. By reducing the software overhead of moving between predictive and generative, scalar and tensorial settings, GEqTrain aims to make equivariant modeling more reproducible, extensible, and reusable.

Daniele Angioletti, Marco Nobile, Vittorio Limongelli

Faculty of Biomedical Sciences, Euler Institute, Universit´a della Svizzera italiana, 6900 Lugano, Switzerland

Flow Matching Generative Modeling Equivariant GNNs Equivariant Representation Learning

Jul 16, 2026cs.SD

MIDI-RAE-JEPA: Hierarchical Representation Learning and Generation for Symbolic Music

Rich internal representations of musical structure are essential for music understanding tasks such as machine-assisted music co-writing, yet self-supervised approaches for symbolic music representation remain underexplored, particularly those that encode the hierarchical multiscale nature of musical structures. We present MIDI-RAE-JEPA, combining a pitch- and time-shift equivariance objective with LeJEPA and a Swin Transformer V2 encoder to learn such hierarchical representations of symbolic music encoded as piano roll images. The time-shift equivariance objective encourages the model to internalize temporal musical relationships. The encoder is trained purely on self-supervised objectives -- including a masked embedding predictor (MEP) -- with collapse prevented via SIGReg. A separate decoder trained on the frozen encoder embeddings achieves reconstruction F1 of 0.995, and a flow matching generative model conditioned on those embeddings produces generations that closely match the pitch register and rhythmic density of the conditioning excerpt, while mismatched conditioning yields unrelated but musically plausible output. Learned representations outperform a Haar scattering transform baseline on a downstream emotion classification task, and embedding distances increase monotonically with pitch and time shift magnitude, confirming measurable equivariance. These results suggest that equivariance-based SSL objectives, combined with sufficient fine-level encoder capacity, provide a viable path toward semantically rich, generatively useful representations of symbolic music.

Scott H. Hawley

Department of Chemistry & Physics, Belmont University, Nashville, TN, USA

Representation Learning Self-Supervised Pre-Training Symbolic Music Generation Hierarchical Representation Learning Equivariant Representation Learning

Jul 7, 2026cs.LG

When Do Geometric Algebra Layers Beat Scalarization? A Controlled Study on SO(3)-Equivariant Vector Laws

Compact networks built from Clifford algebra Cl(3,0) primitives are exactly SO(3)-equivariant and learn synthetic 3D vector laws from few samples. We ask whether the geometric algebra structure itself contributes anything beyond exact equivariance. We compare against a minimal scalarization baseline: invariant dot products fed to a small MLP that outputs coefficients on the equivariant basis {v_i, v_i x v_j}, which is also exactly equivariant. On single-stage laws (rotation by axis-angle, cross product, central force), scalarization matches or beats the Cl(3,0) network at a fraction of the training cost, so the geometric algebra adds nothing there. On compositional targets whose computation graph nests group operations (apply R2 R1 to a point; map a local force through an orientation, then take a torque), the Cl(3,0) network beats scalarization by an order of magnitude in the low-data regime, reaching with 100 samples what the baseline needs 3000 for, and the gap survives strengthening the baseline with the triple-product invariant and 17x more parameters, external Vector Neurons and e3nn baselines, and a multiplicative coefficient network. Ablations show the required network depth tracks the rotation chain length, and scalarization falls below the constant predictor on chains of four rotations. The advantage is not composition per se: on a rotation-free nested cross product, which flattens into polynomial invariant coefficients, scalarization wins by 24x. No tested model, equivariant or not, extrapolates invariant magnitudes: on radius and separation shifts every model is worse than a constant predictor once errors are normalized. We conclude that geometric algebra layers are not a general shortcut for low-data 3D learning, but become useful precisely when the target composes group elements in depth.

Fabien Polly

Equivariant Neural Networks Equivariant Representation Learning Invariant Representation Learning

Jul 4, 2026cs.LG

Foundations of Equivariant Deep Learning: Unifying Graph and Sheaf Neural Networks

Symmetry is everywhere in nature and society. Geometric deep learning exploits symmetries in data to improve the performance and efficiency of deep learning systems. In this paper, we extend geometric deep learning to utilize richer symmetry structures. Specifically, we develop order-equivariant neural networks (OENN), which generalize standard graph message passing and sheaf neural networks via the theory of equivariant bundles over face posets (face categories). We (i) characterize all linear order-equivariant maps, (ii) build OENN layers, and (iii) prove universal approximation theorems (UATs) for continuous order-equivariant maps, which are new results even when restricted to sheaf neural networks (for which no UAT was known before). We illustrate the framework on graph and sheaf models. Our results can also be seen as extending the known UAT for graph neural networks to a more general setting that subsumes sheaf neural networks as well. In addition, we show that OENN can be extended further to CENN, Category-Equivariant Neural Network, which gives the general form of equivariant neural networks as well as of equivariant universal approximation theorems, allowing us to leverage categorical symmetry in data (e.g., non-invertible symmetries on multiple objects with compositional relations on those symmetries).

Yoshihiro Maruyama

Department of Mathematical and Information Sciences, Kyoto University, Kyoto, Japan.

Graph Neural Networks Equivariant Neural Networks Sheaf Neural Networks Neural Network Approximation Theory Equivariant Representation Learning

Jun 26, 2026cs.CV

A Unified Framework for Vision Transformers Equivariant to Discrete Subgroups of $\mathrm{O}(2)$

Vision transformers have become a dominant architecture for visual recognition. However, standard models do not explicitly encode the planar symmetries that arise in many vision domains. We introduce a family of vision transformers equivariant to arbitrary discrete subgroups of

\mathrm{O}(2)

, providing a unified framework that generalizes prior flipping- and

D_4

-equivariant transformer architectures. Our construction yields equivariant analogues of the core transformer components, together with expressivity guarantees for the resulting layers. In particular, we show that whenever

H \le G

, the class of

G

-equivariant ViTs embeds naturally into the class of

H

-equivariant ViTs. We also prove that, in the single-head setting, the corresponding equivariant self-attention layer realizes every

G

-equivariant self-attention map representable by ordinary self-attention. We further construct a

D_6

-equivariant model based on hexagonal patches, making the architecture compatible with six-fold rotational symmetries. We evaluate the resulting models on the PatternNet aerial image dataset in artificially data-scarce regimes across subgroups of

D_4

and

D_6

. Our experiments compare two equivariant attention mechanisms and analyze how the choice of homogeneous-space configurations used in the nonlinearities affects performance. Preliminary results under matched parameter budgets indicate that equivariance can improve recognition accuracy, motivating further study of how discrete symmetry groups shape transformer-based visual recognition models.

Tīkun Ông, Georg Bökman

Independent Researcher · University of Amsterdam

Vision Transformer Equivariant Neural Networks Equivariant Representation Learning Image Classification

Jun 4, 2026cs.LG

Equivariant Neural Belief Propagation

Probabilistic inference over spatially embedded variables requires beliefs that respect

SE(3)

symmetry, yet existing equivariant networks produce only scalars and vectors -- not the rank-2 precision tensors needed for anisotropic uncertainty, and single-component messages collapse multi-modal energy landscapes to physically meaningless averages. We introduce Equivariant Neural Belief Propagation (ENBP), a factor-graph framework whose messages are equivariant Gaussian mixture models with sufficient statistics that transform exactly under

SE(3)

. Rank-2 precision matrices are synthesised via equivariant outer products, ingested through differentiable spectral decomposition, and kept tractable by a greedy KL-based mixture reduction that provably commutes with

SE(3)

. On GEOM-QM9 and GEOM-Drugs, ENBP achieves 98.9% conformational coverage at 0.090

\mathring{A}

error with sub-second latency -- over

100\times

faster than diffusion baselines at higher accuracy. On multi-body robotic inference, vanilla loopy BP diverges at 15+ agents while ENBP converges with near-zero collision rates and machine-precision equivariance error (

{\sim}10^{-7}

vs.\

10^{-1}

for augmented baselines).

Zehua Cheng, Wei Dai, Jiahao Sun

Department of Computer Science, University of Oxford, Oxford, United Kingdom · FLock.io, London, United Kingdom

Factor Graph Optimization Gaussian Mixture Models Bayesian Inference Equivariant Neural Networks Probabilistic Graphical Models Equivariant Representation Learning

Jun 2, 2026cs.LG

Exact equivariance, kept through training, buys zero-shot generalisation across the symmetry group

A latent world model built from an equivariant encoder and predictor inherits a provable symmetry of its training loss: when the dynamics carries a group

G

acting on latents by an orthogonal representation

ρ(g)

, the one-step prediction relMSE is exactly invariant across the whole group, so fitting a restricted slice of orientations mathematically determines it on the entire orbit. The symmetry survives a real Muon/AdamW+EMA+VICReg run -- composed residual

\sim 10^{-6}

after training, under any optimiser (intrinsic Vector-Neuron/e3nn parametrisation) -- and one-step error is flat across the group (5-seed medians: equivariant

\times 1.00

vs a higher-capacity non-equivariant baseline

\times 12.7

in 2D,

\times 17.2

in 3D), while that baseline fits the slice but breaks out-of-distribution. The flatness is not a synthetic artefact: on real-robot DROID end-effector trajectories the equivariant model stays flat across the orbit (

\times 1.000

, rotation residual

1.5\times 10^{-16}

) while a

4.5\times

-larger baseline degrades

\times 11

. One caution is load-bearing: flatness is necessary, not sufficient -- the theorem transports the in-distribution error level unchanged but does not lower it (3D relMSE

\approx 0.43

): across-group error is constant, not low. The same isometry lifts to a closed-loop corollary: under a matching equivariant planner the control error is invariant across the group -- float-floor-exact in 2D/SO(2), statistically flat in 3D/SE(3). Stress-tested against Sutton's Bitter Lesson (augmentation, scale, soft-equivariance), each closes at most the across-group task metric, never the float-floor exactness. This is the generalisation-side foundation of a certified-world-models programme (arXiv:2606.13092, 2606.24945, 2606.24946): flatness transports competence, and the trust bounds built on it are downstream products.

Hongbo Wang

Department of Mathematics, Stony Brook University, Stony Brook, NY 11794, USA

World Models for Robotics World Models Equivariant Neural Networks Latent World Models Equivariant Representation Learning

May 29, 2026cs.RO

SSR: Scaling Surefooted and Symmetric Humanoid Traversal to the Open World

Extending humanoid traversal to the open world is key to practical deployment in human environments, but remains challenging. The robot must use vision to ensure safe and reliable foot placement on heterogeneous terrain under highly dynamic motion, while producing coordinated, natural whole-body behaviors. We propose SSR, an efficient end-to-end framework for egocentric vision-based humanoid traversal that jointly learns these capabilities. SSR introduces imagined foothold guidance, which learns to model forthcoming swing-foot contacts and evaluates their support to guide pre-touchdown swings toward stable regions, reducing edge slips. It further employs equivariant latent-space symmetry augmentation to efficiently induce bilateral coordination under high-dimensional visual observations, and uses terrain-specific multi-discriminator motion priors to encourage human-like behavior across scenes. Extensive experiments show that SSR achieves safe, stable, and high-quality locomotion on diverse real-world terrains, including stairs with varied structures and extreme challenges such as wide gaps and high platforms, while enabling reliable long-horizon traversal in open outdoor environments.

Ruiqi Yu, Yiwen Wang, Yuan Hao +2

Zhejiang University

Humanoid Robot Locomotion Terrain-Aware Robot Locomotion Robot Locomotion Equivariant Representation Learning Vision-Based Robot Control

May 26, 2026cond-mat.soft

On the Equivariant Learning of the $Q$ -tensor Order Parameter

We construct and evaluate group-equivariant neural networks for the prediction of the two-dimensional

Q

-tensor order parameter of nematic liquid crystals from synthetically generated microscopic textures. Seven architectures, equivariant to cyclic groups

C_k

of order

k

for

k=4,\,8,\,16,\,32,\,64,\,128,\, 256

, are built using a combination of weight-sharing constraints, equivariant activations and regularization techniques. To do this, we construct rotation-like permutation matrix groups with elements

\varrho_{C_k}(g)

that act on row-wise vectorized images, thereby approximating a

\frac{2π}{k}

rotation of the circular subdomain on square images. We show that all seven equivariant models satisfy the

Q

-tensor equivariance constraint to within single-precision floating point accuracy. Comparing against approximate parameter-matched non-equivariant benchmarks, with and without data augmentation, we find that the equivariant models consistently achieve lower errors and generalize more robustly to unseen defect configurations. Performance increases with group order, suggesting that the incorporation of finer rotational symmetry leads to lower errors.

Julia Navarro, Mark Wilkinson

Nottingham Trent University, UK · Berea College, USA

Equivariant Neural Networks Materials Science Equivariant Representation Learning

May 26, 2026cs.LG

How the Optimizer Shapes Learned Solutions in Equivariant Neural Networks

Equivariant neural networks encode geometric symmetries by construction, yet they are often difficult to optimize and can underperform less constrained architectures. A growing body of work addresses this through architectural modifications such as constraint relaxation or approximate equivariance, while the role of the optimizer remains comparatively underexplored. We study this direction by comparing Muon and Adam across several equivariant and geometric architectures under pointcloud and molecular learning settings. On ModelNet40, where the comparison is clearest, Muon consistently improves over Adam across all architectures considered. We then analyze the trained ModelNet40 checkpoints through Hessian estimates, loss surface visualizations, and spectral properties of learned weights and intermediate representations. The checkpoints reached by Muon have larger Hessian curvature summaries but more regular loss surfaces, and their learned weights and representations have higher stable and effective ranks. These observations suggest that the interaction between optimizer design and geometric inductive bias deserves further attention from the community.

Teodor-Mihai Stupariu, Andrei Manolache

1Bitdefender, Romania · 2Tudor Vianu High School of Com · University of Stuttgart, Germany +1

Neural Representation Geometry Neural Network Optimization Equivariant Neural Networks Muon Optimizer Equivariant Representation Learning Neural Network Training Dynamics

May 19, 2026cs.LG

Group-Algebraic Tensors: Provably-optimal Equivariant Learning and Physical Symmetry Discovery

We introduce the

\star_G

tensor algebra, in which any finite group

G

defines the multiplication rule, making equivariance an intrinsic algebraic property rather than an architectural constraint. The framework rests on three machine-verified theoretical pillars: (i)~an Eckart-Young optimality guarantee for the

\star_G

-SVD: the first such result for symmetry-preserving tensor approximation, exact and polynomial-time; (ii)~a Kronecker factorization that composes multiple symmetries by replacing

F_G

with

F_{G_1} \otimes F_{G_2}

with no architectural redesign; and (iii)~~a 600-line Lean~~4 formalization of the

\star_G

algebra. The framework provides capabilities that equivariant neural networks (ENNs) structurally cannot: a closed-form per-irreducible-representation decomposition of every prediction, and data-driven discovery of the symmetry group that best fits a dataset. As a non-trivial empirical demonstration, decomposing QM9 molecular geometry over the chiral octahedral subgroup of SO(3) recovers the Wigner--Eckart selection rules of angular momentum from data alone, with no quantum mechanical input: scalar properties are A

_1

-dominated, dipole components are T

_1

-dominated, the isotropic polarizability is uniquely insensitive to

l\!=\!1

as the rank-2-trace decomposition

l\!=\!0 \oplus l\!=\!2

requires, and the T

_1

/A

_1

predictive-power ratio separates vector observables from scalar observables by a factor of five. On full QM9 (130{,}831 molecules),

\star_G

-SVD with ridge regression provides closed form predictions at

\sim50-90\times

fewer parameters than parameter-matched MLPs. Algebraic equivariance thus complements architectural equivariance not as a faster-better-cheaper alternative but as a different mathematical affordance: provably-optimal symmetry-preserving compression, per-irrep interpretability, and data-driven physical discovery.

Paulina Hoyos, Shashanka Ubaru, Dongsung Huh +5

UT Austin · IBM Research · Independent +2

Scientific Discovery Equivariant Neural Networks Tensor Decomposition Equivariant Representation Learning Singular Value Decomposition

May 5, 2026cs.SD

PHALAR: Phasors for Learned Musical Audio Representations

Stem retrieval, the task of matching missing stems to a given audio submix, is a key challenge currently limited by models that discard temporal information. We introduce PHALAR, a contrastive framework achieving a relative accuracy increase of up to

\approx 70\%

over the state-of-the-art while requiring

<50\%

of the parameters and a 7

\times

training speedup. By utilizing a Learned Spectral Pooling layer and a complex-valued head, PHALAR enforces pitch-equivariant and phase-equivariant biases. PHALAR establishes new retrieval state-of-the-art across MoisesDB, Slakh, and ChocoChorales, correlating significantly higher with human coherence judgment than semantic baselines. Finally, zero-shot beat tracking and linear chord probing confirm that PHALAR captures robust musical structures beyond the retrieval task.

Davide Marincione, Michele Mancusi, Giorgio Strano +4

Department of Computer Science, Sapienza University of Rome, Italy · Moises Systems, Inc. · Paradigma, Inc.

Audio Representation Learning Representation Learning Audio Understanding Music Information Retrieval Equivariant Representation Learning

Jul 31, 2025cs.LG

L-GTA: Latent Generative Modeling for Time Series Augmentation

Data augmentation is becoming increasingly important across various areas of time series analysis, including forecasting, classification, and anomaly detection. We introduce the Latent Generative Temporal Augmentation (L-GTA) model, a generative approach based on a Variational Autoencoder with a Bi-LSTM backbone and temporal self-attention. The model learns a latent representation for each timestep and applies controlled perturbations such as jittering, magnitude warping, or drift. We define an equivariance objective to further encourage consistency between latent space and data space transformations. As a result, the augmented samples show predictable and interpretable transformation signatures. We evaluate L-GTA on several real-world datasets against SOTA generative methods, including TimeGAN, TimeVAE, and Diffusion-TS, as well as direct transformation approaches. Across experiments on downstream forecasting, distribution fidelity, and controllability of transformation intensity, L-GTA consistently outperforms competing approaches. In downstream forecasting, it reduces prediction error by up to 26% compared to the strongest generative method and 27% relative to using the original data without augmentation.

Luis Roque, Vitor Cerqueira, Carlos Soares +1

Faculdade de Engenharia da Universidade do Porto, Porto, Portugal · University of Coimbra, Coimbra, Portugal · Fraunhofer Portugal AICOS, Portugal +1

Variational Autoencoders Synthetic Data Augmentation Representation Learning Time Series Data Augmentation Equivariant Representation Learning

May 26, 2025cs.LG

Representation Learning for Equivariant Inference with Guarantees

In many real-world applications of regression, conditional probability estimation, and uncertainty quantification, exploiting symmetries rooted in physics or geometry can dramatically improve generalization and sample efficiency. While geometric deep learning has made empirical advances by incorporating symmetry and geometry priors, less attention has been given to statistical learning guarantees. In this paper, we introduce an equivariant representation learning framework that simultaneously addresses regression, conditional probability estimation, and uncertainty quantification while providing first-of-its-kind non-asymptotic statistical learning guarantees. Grounded in operator and group representation theory, our framework approximates the spectral decomposition of the conditional expectation operator, building representations that are both equivariant and disentangled along independent symmetry quotient groups. Empirical evaluations on synthetic datasets and real-world robotics applications confirm the potential of our approach, matching or outperforming existing equivariant baselines in regression while providing well-calibrated uncertainty estimates.

Daniel Ordoñez-Apraez, Vladimir Kostić, Alek Fröhlich +3

CSML, Italian Institute of Technology, Genova, Italy · Universit`a di Genova, Genova, Italy · University of Novi Sad, Novi Sad, Serbia +2

Disentangled Representation Learning Representation Learning Uncertainty Quantification Conditional Distribution Estimation Equivariant Representation Learning Statistical Learning Theory