Positional Encoding is All You Need for Scalable Equivariance Constraint Relaxation
Organizations: Technion – Israel Institute of Technology · NVIDIA
Abstract
Equivariant neural networks incorporate known symmetries of the data, such as permutations, rotations, or translations, directly into their architecture, often improving sample efficiency and generalization. However, strict equivariance can make optimization difficult, for example, when the data only approximately satisfy the assumed symmetry, or when the equivariance constraints induce a challenging loss landscape. Recent studies have shown that relaxing equivariance during training by introducing non-equivariant dense linear layers can ease optimization in such cases and improve the performance of the resulting strictly equivariant model, after removing the relaxation layers at test time. While effective, this approach incurs a significant computational overhead, limiting its applicability in scalable architectures. In this work, we propose a more efficient constraint-relaxation method based on positional encoding (PE). Our framework injects a positional encoding with learnable scale that breaks the model's original symmetry. This positional signal is gradually annealed to zero, recovering the original equivariant model by the end of training. Empirically, we demonstrate significant improvements for translational and rotational equivariant models with only marginal compute and memory overhead.
Figures & tables
| Model | Instance Accuracy | Parameters | Time per Epoch [sec] |
|---|---|---|---|
| VN-PointNet [ 10 ] | 67.5 0.5 | 1,972,745 | 27.6 |
| VN-PointNet-CR [ 1 ] | 74.0 1.9 | 8,978,126 | 97.6 |
| VN-PointNet-PE add. (ours) | 79.6 0.4 | 1,972,746 | 27.9 |
| VN-PointNet-PE concat. (ours) | 79.3 0.9 | 1,972,872 | 27.7 |
| VN-DGCNN [ 10 ] | 88.3 0.2 | 2,899,830 | 61.2 |
| VN-DGCNN-CR [ 1 ] | 88.6 0.5 | 8,829,555 | 181.7 |
| Model | Accuracy | Random-Rotation Accuracy |
|---|---|---|
| Rapidash-Eq [ 30 ] | 83.7 1.4 | 83.6 1.0 |
| Rapidash-Symm. Break [ 30 ] | 88.0 0.1 | 14.8 1.1 |
| Rapidash-CR [ 1 ] | 84.3 0.8 | 84.2 0.5 |
| Rapidash-Eq+PE (ours) | 84.6 0.6 | 84.6 0.5 |
| Model | Accuracy | Shift accuracy | Change (%) |
|---|---|---|---|
| XCiT-Nano [ 48 ] | 70.5 0.2 | 68.8 0.0 | -2.4 |
| AF-XCiT-Nano [ 31 ] | 70.6 0.1 | 70.6 0.1 | -0.0 |
| AF-XCiT-Nano + PE (train and test) | 71.4 0.1 | 70.3 0.1 | -1.5 |
| AF-XCiT-Nano + PE (train only; ours) | 71.1 0.1 | 71.1 0.1 | -0.0 |
| Model | Future | Domain |
|---|---|---|
| E2CNN [ 32 ] | 81.2 1.1 | 59.8 2.0 |
| RSteer [ 13 ] | 79.4 0.1 | 58.0 0.1 |
| E2CNN + PE (train and test) | 79.6 1.2 | 58.5 0.7 |
| E2CNN + PE (train only; ours) | 79.8 0.8 | 58.2 0.7 |
| Model | Test MSE ( ) |
|---|---|
| SEGNN [ 33 ] | 44.3 1.4 |
| SEGNN-CR [ 1 ] | 43.2 1.9 |
| SEGNN-PE (Ours) | 41.9 1.5 |
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
| Annealing Schedule | Reg. | PE learnable scale | Accuracy equivariant (w/o PE) | Accuracy (with PE) | |
|---|---|---|---|---|---|
| 1 | None | 0 | ✗ | 4.5 0.3 | 35.6 0.3 |
| 2 | None | 0 | ✓ | 79.7 2.7 | 79.8 2.1 |
| 3 | None | 0.001 | ✗ | 4.5 0.3 | 35.6 0.3 |
| 4 | None | 0.001 | ✓ | 80.7 0.8 | 80.8 0.9 |
| 5 | Linear | 0 | ✗ | 50.9 0.7 | 51.5 0.3 |
| 6 | Linear | 0 | ✓ | 80.4 1.0 | 80.4 0.9 |
| Annealing Schedule | Reg. | PE learnable scale | Accuracy equivariant (w/o PE) | Accuracy (with PE) | |
|---|---|---|---|---|---|
| 1 | None | 0 | ✗ | 74.1 3.6 | 88.8 0.1 |
| 2 | None | 0 | ✓ | 88.6 0.2 | 88.6 0.1 |
| 3 | None | 0.001 | ✗ | 74.1 3.6 | 88.8 0.1 |
| 4 | None | 0.001 | ✓ | 88.5 0.4 | 88.5 0.3 |
| 5 | Triangular | 0 | ✗ | 88.1 0.5 | 88.6 0.4 |
| 6 | Triangular | 0 | ✓ | 88.2 0.1 | 88.2 0.1 |
| Model | PE variant | Val accuracy | |
|---|---|---|---|
| VN-PointNet | Fixed-PE concat | ||
| VN-PointNet | FixedSO-PE concat | ||
| VN-DGCNN | Fixed-PE concat | ||
| VN-DGCNN | FixedSO-PE concat |
| Method | MSE ( ) |
|---|---|
| Linear [ 9 ] | 819 |
| SE(3)-Tr. [ 58 ] | 244 |
| RF [ 59 ] | 104 |
| ClofNet [ 60 ] | 65 |
| EGNN [ 9 ] | 71 |
| EGNO [ 61 ] | 54 |
| Integration | Input only | Every layer |
|---|---|---|
| Additive PE | 79.6 0.4 | 79.3 1.6 |
| Concatenative PE | 79.3 0.9 | 78.7 0.9 |
| Model | 100% | 75% | 50% | 25% |
|---|---|---|---|---|
| VN-PointNet (Baseline) | 67.5 0.5 | 64.5 0.2 | 62.3 1.1 | 48.6 3.4 |
| VN-PointNet-PE add. (ours) | 79.6 0.4 | 77.2 1.2 | 68.8 4.3 | 55.1 2.2 |
| VN-PointNet-PE concat. (ours) | 79.3 0.9 | 75.5 0.5 | 72.0 1.2 | 53.1 4.4 |
| Hyperparameter | Value |
|---|---|
| Dataset | ModelNet40 |
| Input Points | 300 |
| Batch Size | 32 |
| Optimizer | Adam (VN-PointNet) / SGD (VN-DGCNN) |
| Learning Rate | 0.001 (VN-PointNet) / 0.1 (VN-DGCNN) |
| Weight Decay |
| Hyperparameter | Method | Values Tested |
|---|---|---|
| Positional Encoding Type | PE (Ours) | {Sinusoidal, Fixed} |
| Mix-coef Scheduler | PE / CR | {Linear, Triangular} |
| Regularization | PE / CR | {0.1, 0.05, 0.01, 0.005, 0.001} |
| Model | PE Variant | PE Type | Scheduler | Regularization |
|---|---|---|---|---|
| VN-PointNet | Additive | Sinusoidal | Linear | 0.001 |
| VN-PointNet | Concat | Sinusoidal | Linear | 0.005 |
| VN-DGCNN | Additive | Fixed | Linear | 0.001 |
| VN-DGCNN | Concat | Sinusoidal | Triangular | 0.001 |
| Hyperparameter | Value |
|---|---|
| Dataset | ModelNet40 |
| Input Points | 1024 |
| Task | Object classification |
| Hidden Features | 256 |
| Max Epochs | 120 |
| Rapidash Layers | 7 |
| Hyperparameter | Method | Values Tested |
|---|---|---|
| Positional Encoding Type | PE (Ours) | {Sinusoidal, Fixed} |
| Mix-coef Scheduler | PE / CR | {Linear, Triangular} |
| Regularization | PE / CR | {0.1, 0.05, 0.01, 0.005, 0.001} |
| Method | PE Variant | PE Type | Scheduler | Regularization |
|---|---|---|---|---|
| Rapidash-Eq+PE | Additive | Sinusoidal | Linear | 0.05 |
| Rapidash-CR | – | – | Triangular | 0.01 |
| Hyperparameter | Value |
|---|---|
| Architecture | AF-XCiT-Nano |
| Total Epochs | 400 |
| Relaxation Decay Epochs | 320 |
| Strict Equivariance Epochs | 80 |
| Regularization | |
| Batch Size | 512 |
| Hyperparameter | Values Tested |
|---|---|
| Regularization | |
| Decay Schedule | {Linear, Triangular} |
| Hyperparameter | Value |
|---|---|
| Dataset | Smoke plumes (Rotation) |
| Task | Velocity-field forecasting |
| Symmetry Group | |
| Layers | 5 |
| Hidden Channels | 92 |
| Kernel Size |
| Hyperparameter | Values Tested |
|---|---|
| Mix-coef Scheduler | {Linear, Triangular} |
| Schedule Length (epochs) | {50, 100} |
| Regularization | {0.1, 0.05, 0.01, 0.005, 0.001} |
| Variant | Scheduler | Schedule Length | Regularization |
|---|---|---|---|
| Decayed PE | Linear | 50 | 0.005 |
| Persistent PE | None | – | 0 |
| Hyperparameter | Value |
|---|---|
| Dataset | Charged-particle N-body |
| Split (Train/Val/Test) | 3000 / 2000 / 2000 |
| Particles | 5 |
| Batch Size | 100 |
| Max Epochs | 2000 |
| Optimizer | Adam |
| Hyperparameter | Values Tested |
|---|---|
| Learning Rate | |
| Weight Decay |
| Hyperparameter | Method | Values Tested |
|---|---|---|
| Mix-coef Scheduler | PE / CR | {Linear, Triangular} |
| Regularization | PE / CR | {0.1, 0.05, 0.01, 0.005, 0.001} |
| PE Type | PE (Ours) | {Sinusoidal, Learned} |
| Method | Scheduler | Regularization | PE Type |
|---|---|---|---|
| CR | Linear | 0.1 | - |
| PE (Ours) | Triangular | 0.05 | Learned |
Equivariant Neural Networks
Equivariant neural networks encode transformations such as rotations, translations, and permutations so model outputs change consistently with their inputs.
See all 165 papers