Equivariant neural networks incorporate known symmetries of the data, such as permutations, rotations, or translations, directly into their architecture, often improving sample efficiency and generalization. However, strict equivariance can make optimization difficult, for example, when the data only approximately satisfy the assumed symmetry, or when the equivariance constraints induce a challenging loss landscape. Recent studies have shown that relaxing equivariance during training by introducing non-equivariant dense linear layers can ease optimization in such cases and improve the performance of the resulting strictly equivariant model, after removing the relaxation layers at test time. While effective, this approach incurs a significant computational overhead, limiting its applicability in scalable architectures. In this work, we propose a more efficient constraint-relaxation method based on positional encoding (PE). Our framework injects a positional encoding with learnable scale that breaks the model's original symmetry. This positional signal is gradually annealed to zero, recovering the original equivariant model by the end of training. Empirically, we demonstrate significant improvements for translational and rotational equivariant models with only marginal compute and memory overhead.
Figures & tables
Figure 1: Equivariance constraint relaxation during training. Left: A strictly equivariant model, whose hypothesis class is restricted to equivariant functions. Middle: Dense constraint relaxation [ 1 , 2 , 3 ] adds a non-equivariant dense branch in parallel to every equivariant layer during training. This greatly enlarges the hypothesis class, but substantially increases the parameter count and computational cost of training. Right: Our PE constraint relaxation instead adds a single positional encoding to the input. This relaxes the constraint with negligible parameter and compute overhead. In both relaxation methods, the auxiliary component is used only during training and is annealed to zero, recovering the original strictly equivariant architecture at inference.
Model
Instance Accuracy
Parameters
Time per Epoch [sec]
VN-PointNet [ 10 ]
67.5 ± 0.5
1,972,745
27.6
VN-PointNet-CR [ 1 ]
74.0 ± 1.9
8,978,126
97.6
VN-PointNet-PE add. (ours)
79.6 ± 0.4
1,972,746
27.9
VN-PointNet-PE concat. (ours)
79.3 ± 0.9
1,972,872
27.7
VN-DGCNN [ 10 ]
88.3 ± 0.2
2,899,830
61.2
VN-DGCNN-CR [ 1 ]
88.6 ± 0.5
8,829,555
181.7
Table 1: ModelNet40 classification accuracy and training cost. Results for VN-PointNet and VN-DGCNN under strict equivariance, dense-layer constraint relaxation (CR), and our positional-encoding relaxation (PE; additive and concatenative variants). Accuracy is the average best validation accuracy over training (mean ± std over 5 random seeds). All models are trained and evaluated on randomly rotated point clouds ( SO(3)/SO(3) ). “Parameters” counts trainable parameters in the trained architecture, and “Time per epoch” is computed by dividing total wall-clock training time on a single NVIDIA RTX A6000 with batch-size 32 by the number of epochs.
Model
Accuracy
Random-Rotation Accuracy
Rapidash-Eq [ 30 ]
83.7 ± 1.4
83.6 ± 1.0
Rapidash-Symm. Break [ 30 ]
88.0 ± 0.1
14.8 ± 1.1
Rapidash-CR [ 1 ]
84.3 ± 0.8
84.2 ± 0.5
Rapidash-Eq+PE (ours)
84.6 ± 0.6
84.6 ± 0.5
Table 2: ModelNet40 classification with Rapidash. We report validation accuracy on the original validation set (“Accuracy”) and on the validation set under random rotations (“Random-Rotation Accuracy”). PE improves both standard accuracy and rotated accuracy over the strictly equivariant Rapidash baseline, while preserving the original equivariant test-time architecture. In contrast, Rapidash-Symm. Break achieves the highest standard accuracy but collapses under random rotations, indicating strong reliance on the orientation bias present in the dataset.
Model
Accuracy
Shift accuracy
Change (%)
XCiT-Nano [ 48 ]
70.5 ± 0.2
68.8 ± 0.0
-2.4
AF-XCiT-Nano [ 31 ]
70.6 ± 0.1
70.6 ± 0.1
-0.0
AF-XCiT-Nano + PE (train and test)
71.4 ± 0.1
70.3 ± 0.1
-1.5
AF-XCiT-Nano + PE (train only; ours)
71.1 ± 0.1
71.1 ± 0.1
-0.0
Table 3: ImageNet results for AF-XCiT-Nano. We report ImageNet validation top-1 accuracy (“Accuracy”), accuracy under random cyclic translations (“Shift Accuracy”), and the relative change between the two. We report mean ± std over 2 random seeds. AF-XCiT + PE (train and test) keeps positional encoding at inference and therefore corresponds to a relaxed, only approximately invariant setting, whereas AF-XCiT + PE (train only; ours) anneals the PE coefficient to zero during training and removes PE at test time, recovering the original alias-free invariant architecture. The results show that the annealed variant preserves substantially better accuracy under input translations than keeping PE at test time, while still improving accuracy over the strict AF-XCiT baseline.
Model
Future ↓
Domain ↓
E2CNN [ 32 ]
81.2 ± 1.1
59.8 ± 2.0
RSteer [ 13 ]
79.4 ± 0.1
58.0 ± 0.1
E2CNN + PE (train and test)
79.6 ± 1.2
58.5 ± 0.7
E2CNN + PE (train only; ours)
79.8 ± 0.8
58.2 ± 0.7
Table 4: Smoke plume forecasting with E2CNN. We report 20-step prediction RMSE ( ×10−2 , lower is better) on the Future and Domain test sets of Wang et al. [13] (mean ± std over 3 seeds). E2CNN + PE (train and test) keeps the PE at inference and is therefore only approximately equivariant, similar to RSteer, whereas E2CNN + PE (train only; ours) anneals the PE to zero during training and recovers the strictly equivariant E2CNN at inference.
Model
Test MSE ( ×10−4 ) ↓
SEGNN [ 33 ]
44.3 ± 1.4
SEGNN-CR [ 1 ]
43.2 ± 1.9
SEGNN-PE (Ours)
41.9 ± 1.5
Table 5: N-Body (charged particles) forecasting with SEGNN. We report test MSE on final particle positions evaluated at the checkpoint with the lowest validation loss (mean ± std over 5 seeds).
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
Annealing Schedule λ
Reg. β
PE learnable scale γ
Accuracy equivariant (w/o PE)
Accuracy (with PE)
1
None
0
✗
4.5 ± 0.3
35.6 ± 0.3
2
None
0
✓
79.7 ± 2.7
79.8 ± 2.1
3
None
0.001
✗
4.5 ± 0.3
35.6 ± 0.3
4
None
0.001
✓
80.7 ± 0.8
80.8 ± 0.9
5
Linear
0
✗
50.9 ± 0.7
51.5 ± 0.3
6
Linear
0
✓
80.4 ± 1.0
80.4 ± 0.9
Appendix
Table 6: Ablation study for VN-PointNet on ModelNet40. We vary the annealing schedule, regularization strength, and whether the PE scale is learnable. Results are mean ± std over 3 seeds. Training and evaluation use randomly rotated point clouds ( SO(3)/SO(3) ).
Annealing Schedule λ
Reg. β
PE learnable scale γ
Accuracy equivariant (w/o PE)
Accuracy (with PE)
1
None
0
✗
74.1 ± 3.6
88.8 ± 0.1
2
None
0
✓
88.6 ± 0.2
88.6 ± 0.1
3
None
0.001
✗
74.1 ± 3.6
88.8 ± 0.1
4
None
0.001
✓
88.5 ± 0.4
88.5 ± 0.3
5
Triangular
0
✗
88.1 ± 0.5
88.6 ± 0.4
6
Triangular
0
✓
88.2 ± 0.1
88.2 ± 0.1
Appendix
Table 7: Ablation study for VN-DGCNN on ModelNet40. We repeat the same ablation over the annealing schedule, regularization strength, and learnable PE scale. Results are mean ± std over 3 seeds. Training and evaluation use randomly rotated point clouds ( SO(3)/SO(3) ).
Model
PE variant
StabSO(3)
Val accuracy
VN-PointNet
Fixed-PE concat
SO(2)
69.4±1.3
VN-PointNet
FixedSO-PE concat
{I}
70.5±1.6
VN-DGCNN
Fixed-PE concat
SO(2)
88.4±0.2
VN-DGCNN
FixedSO-PE concat
{I}
88.4±0.3
Appendix
Table 8: Fixed-PE stabilizer ablation on Vector Neurons. We compare the original broadcast Fixed-PE anchor, whose SO(3) stabilizer is SO(2) , with the FixedSO-PE frame variant, whose SO(3) stabilizer is trivial. All runs use concat-based PE and the best hyperparameters found for concat-based Fixed-PE in the original sweep. Results are validation accuracy, reported as mean ± std over 3 seeds.
Figure 2: Additional model-width scaling results on VN-PointNet (ModelNet40). We vary model width by multiplying all channel dimensions by the same factor and plot results against the maximum channel dimension (x-axis). (a) Best validation accuracy (mean ± std over 3 seeds). (b) Training time per epoch on a single RTX A6000. (c) Trainable parameters during training. PE consistently improves the equivariant baseline in accuracy while tracking its training cost, whereas CR’s gains come with rapidly increasing overhead at larger widths.
Figure 3: Additional input-size scaling results on VN-PointNet (ModelNet40). We vary the number of points per point cloud while keeping the architecture fixed. (a) Best validation accuracy (mean ± std over 3 seeds). (b) Training time per epoch on a single RTX A6000. All methods exhibit approximately linear scaling in epoch time with the number of points, while PE consistently exceeds the accuracy of CR at similar cost to the equivariant baseline.
Figure 4: Quadratic fits for VN-PointNet width scaling. We report least-squares quadratic fits to epoch time (left) and trainable parameters (right) for the strictly equivariant baseline, dense-layer constraint relaxation (CR), and positional-encoding relaxation (PE).
Figure 5: Training dynamics on VN-PointNet (ModelNet40). We compare the strict equivariant baseline, dense-layer constraint relaxation (CR), and positional-encoding relaxation (PE). (a) Training loss over epochs. (b) Validation accuracy over epochs. Validation accuracy is evaluated with PE removed. PE descends faster and reaches lower training loss through most of training, while also achieving higher validation accuracy after the initial training phase. This suggests that the improvement of PE over both the strict baseline and CR is associated with more favorable optimization dynamics.
Figure 6: Training dynamics on ImageNet for XCiT and AF-XCiT. We compare standard XCiT, strict AF-XCiT, and AF-XCiT with train-time PE that is annealed to zero before inference. (a) Training loss over epochs. (b) Validation accuracy over epochs. Validation accuracy is evaluated with PE removed. The strict AF-XCiT model exhibits a noisier optimization trajectory, whereas AF-XCiT + PE produces smoother training curves and finishes with slightly improved validation accuracy. The PE trajectory remains close to that of standard XCiT while still recovering the original alias-free architecture at test time.
Figure 7: Adversarial cyclic-shift robustness on ImageNet for AF-XCiT. We report validation accuracy under adversarial cyclic translations, which correspond to the symmetry group modeled by AF-XCiT. AF-XCiT + PE (train and test) keeps positional encoding at inference and is therefore only approximately invariant, leading to a substantially larger degradation under this group-aligned attack. AF-XCiT + PE (train only) anneals the positional encoding to zero and recovers the original invariant architecture, closely matching the robustness of strict AF-XCiT.
Figure 8: Adversarial practical-translation robustness on ImageNet for AF-XCiT. We report validation accuracy under crop-shift and bilinear-shift attacks. These transformations reflect practical translations but are not exact elements of the cyclic translation group modeled by AF-XCiT because they introduce boundary, cropping, or interpolation effects. Accordingly, the strict invariant and relaxed variants behave more similarly than in the cyclic-shift setting, suggesting that exact cyclic-shift invariance primarily protects against transformations aligned with the modeled symmetry group.
Figure 9: Adversarial rotation robustness on VN-PointNet (ModelNet40). We compare the relaxed VN-PointNet + PE model evaluated with PE at test time against the projected equivariant model obtained by removing PE at test time. Attack severity corresponds to the number of random rotations available to the adversary. The projected equivariant model is consistently more robust under adversarial rotations, showing that recovering strict equivariance at inference can improve robustness even when the relaxed model is competitive under standard validation.
Method
MSE ( ×10−4 ) ↓
Linear [ 9 ]
819
SE(3)-Tr. [ 58 ]
244
RF [ 59 ]
104
ClofNet [ 60 ]
65
EGNN [ 9 ]
71
EGNO [ 61 ]
54
Appendix
Table 9: N-body forecasting with SEGNN under ACE optimization. We report test MSE on final particle positions (lower is better), following the ACE reporting convention. Results for our runs are mean ± std over 5 seeds. Replacing the ACE MLP relaxation with a learnable per-layer PE improves over the strict SEGNN baseline and ACE-Linear, while remaining slightly behind the original ACE-MLP relaxation. Input-only PE does not benefit from ACE optimization.
Integration
Input only
Every layer
Additive PE
79.6 ± 0.4
79.3 ± 1.6
Concatenative PE
79.3 ± 0.9
78.7 ± 0.9
Appendix
Table 10: PE placement in VN-PointNet (ModelNet40). We report accuracy (mean ± std over 5 seeds) for PE injected only at the input and at every equivariant layer, following the protocol of Table 1 . Injecting PE at every layer does not improve over input-only PE.
Model
100%
75%
50%
25%
VN-PointNet (Baseline)
67.5 ± 0.5
64.5 ± 0.2
62.3 ± 1.1
48.6 ± 3.4
VN-PointNet-PE add. (ours)
79.6 ± 0.4
77.2 ± 1.2
68.8 ± 4.3
55.1 ± 2.2
VN-PointNet-PE concat. (ours)
79.3 ± 0.9
75.5 ± 0.5
72.0 ± 1.2
53.1 ± 4.4
Appendix
Table 11: Data efficiency of VN-PointNet with PE (ModelNet40). We report accuracy (mean ± std over 5 seeds) when training on a fraction of the training set. Both PE variants outperform the strict baseline at every training-set size.
Hyperparameter
Value
Dataset
ModelNet40
Input Points
300
Batch Size
32
Optimizer
Adam (VN-PointNet) / SGD (VN-DGCNN)
Learning Rate
0.001 (VN-PointNet) / 0.1 (VN-DGCNN)
Weight Decay
10−4
Appendix
Table 12: General Hyperparameters for Vector Neurons models.
Hyperparameter
Method
Values Tested
Positional Encoding Type
PE (Ours)
{Sinusoidal, Fixed}
Mix-coef Scheduler
PE / CR
{Linear, Triangular}
Regularization β
PE / CR
{0.1, 0.05, 0.01, 0.005, 0.001}
Appendix
Table 13: Hyperparameter Search Space for Point Cloud Classification Models.
Model
PE Variant
PE Type
Scheduler
Regularization β
VN-PointNet
Additive
Sinusoidal
Linear
0.001
VN-PointNet
Concat
Sinusoidal
Linear
0.005
VN-DGCNN
Additive
Fixed
Linear
0.001
VN-DGCNN
Concat
Sinusoidal
Triangular
0.001
Appendix
Table 14: Optimal Relaxation Hyperparameters for Point Cloud Classification (PE Method).
Hyperparameter
Value
Dataset
ModelNet40
Input Points
1024
Task
Object classification
Hidden Features C
256
Max Epochs
120
Rapidash Layers
7
Appendix
Table 15: General Hyperparameters for Rapidash on ModelNet40.
Hyperparameter
Method
Values Tested
Positional Encoding Type
PE (Ours)
{Sinusoidal, Fixed}
Mix-coef Scheduler
PE / CR
{Linear, Triangular}
Regularization β
PE / CR
{0.1, 0.05, 0.01, 0.005, 0.001}
Appendix
Table 16: Hyperparameter Search Space for Rapidash Relaxation (PE and CR).
Method
PE Variant
PE Type
Scheduler
Regularization β
Rapidash-Eq+PE
Additive
Sinusoidal
Linear
0.05
Rapidash-CR
–
–
Triangular
0.01
Appendix
Table 17: Optimal Relaxation Hyperparameters for Rapidash (PE and CR).
Hyperparameter
Value
Architecture
AF-XCiT-Nano
Total Epochs
400
Relaxation Decay Epochs
320
Strict Equivariance Epochs
80
Regularization β
10−5
Batch Size
512
Appendix
Table 18: General Hyperparameters for AF-XCiT (ImageNet).
Hyperparameter
Values Tested
Regularization β
{0.1,0.01,10−3,10−4,10−5}
Decay Schedule
{Linear, Triangular}
Appendix
Table 19: Hyperparameter Search Space for AF-XCiT (Tuned on Stanford Cars).
Hyperparameter
Value
Dataset
Smoke plumes (Rotation)
Task
Velocity-field forecasting
Symmetry Group
C4
Layers
5
Hidden Channels
92
Kernel Size
3×3
Appendix
Table 20: General Hyperparameters for Smoke Plume Forecasting (E2CNN).
Hyperparameter
Values Tested
Mix-coef Scheduler
{Linear, Triangular}
Schedule Length (epochs)
{50, 100}
Regularization β
{0.1, 0.05, 0.01, 0.005, 0.001}
Appendix
Table 21: Hyperparameter Search Space for Smoke Plume PE Relaxation.
Variant
Scheduler
Schedule Length
Regularization β
Decayed PE
Linear
50
0.005
Persistent PE
None
–
0
Appendix
Table 22: Optimal Relaxation Hyperparameters for Smoke Plume PE Relaxation.
Hyperparameter
Value
Dataset
Charged-particle N-body
Split (Train/Val/Test)
3000 / 2000 / 2000
Particles
5
Batch Size
100
Max Epochs
2000
Optimizer
Adam
Appendix
Table 23: General Hyperparameters for SEGNN (N-Body).
Hyperparameter
Values Tested
Learning Rate
{5⋅10−4,8⋅10−4,1⋅10−3,2⋅10−3,5⋅10−3}
Weight Decay
{10−12,10−8,10−6,10−4}
Appendix
Table 24: Optimization Hyperparameter Search Space (SEGNN Strict Baseline).
Hyperparameter
Method
Values Tested
Mix-coef Scheduler
PE / CR
{Linear, Triangular}
Regularization β
PE / CR
{0.1, 0.05, 0.01, 0.005, 0.001}
PE Type
PE (Ours)
{Sinusoidal, Learned}
Appendix
Table 25: Relaxation Hyperparameter Search Space (SEGNN).
Method
Scheduler
Regularization β
PE Type
CR
Linear
0.1
-
PE (Ours)
Triangular
0.05
Learned
Appendix
Table 26: Optimal Relaxation Hyperparameters for SEGNN.
Equivariant neural networks encode transformations such as rotations, translations, and permutations so model outputs change consistently with their inputs.