cs.LGNov 3, 2026

Positional Encoding is All You Need for Scalable Equivariance Constraint Relaxation

Authors: Hagay Michaeli, Haggai Maron, Daniel Soudry

Organizations: Technion – Israel Institute of Technology · NVIDIA

Abstract

Equivariant neural networks incorporate known symmetries of the data, such as permutations, rotations, or translations, directly into their architecture, often improving sample efficiency and generalization. However, strict equivariance can make optimization difficult, for example, when the data only approximately satisfy the assumed symmetry, or when the equivariance constraints induce a challenging loss landscape. Recent studies have shown that relaxing equivariance during training by introducing non-equivariant dense linear layers can ease optimization in such cases and improve the performance of the resulting strictly equivariant model, after removing the relaxation layers at test time. While effective, this approach incurs a significant computational overhead, limiting its applicability in scalable architectures. In this work, we propose a more efficient constraint-relaxation method based on positional encoding (PE). Our framework injects a positional encoding with learnable scale that breaks the model's original symmetry. This positional signal is gradually annealed to zero, recovering the original equivariant model by the end of training. Empirically, we demonstrate significant improvements for translational and rotational equivariant models with only marginal compute and memory overhead.

Figures & tables

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Flash EQ-Linear: Accelerating Equivariant Linear Layers via Group-wise Discrete Fourier Transform

    Jul 23, 2026Zhongchen Zhao, Jixin Wang, Qi Xie +4Efficient Neural Network InferenceEquivariant Neural Networks

  2. Tunable Soft Equivariance with Guarantees

    Mar 27, 2026Md Ashiqur Rahman, Lim Jun Hao, Jeremiah Jiang +2Equivariant Neural Networks