Enforcing Symmetries on Pre-trained ML Models for Physics
Authors: Wenjie Zhou, Valentino F. Foit, David W. Hogg, Soledad Villar
Organizations: Department of Physics and Astronomy, Johns Hopkins University, Baltimore, MD 21218, USA · Center for Cosmology and Particle Physics, Department of Physics, New York University, New York, NY 10003, USA · Max-Planck-Institut für Astronomie, Königstuhl 17, D-69117 Heidelberg, Germany · Center for Computational Astrophysics, Flatiron Institute, 162 Fifth Ave., New York, NY 10010, USA · Department of Applied Mathematics and Statistics and Mathematical Institute for Data Science, Johns Hopkins University, Baltimore, MD 21218, USA · Center for Computational Mathematics, Flatiron Institute, 162 Fifth Ave., New York, NY 10010, USA
Post-hoc group averaging (or test-time augmentation) is a simple way to impose symmetry on a trained predictor without changing its architecture or retraining it. For finite and compact groups, averaging over a normalized Haar measure on the group gives the Reynolds operator: a canonical projection onto invariant or equivariant predictors with known non-increasing risk guarantees under certain assumptions. This paper provides numerical experiments on physics emulation models that demonstrate the effectiveness of the approach in this setting. We then extend this framework to non-compact group symmetries. For non-compact groups, the Haar measure cannot be normalized, so group averaging cannot be defined in the same way. We replace it with three alternatives: the Weyl unitary trick (which requires complexifying the space and extending the predictor), a data-dependent orthogonal projection (which involves a non-trivial optimization on the space of invariant or equivariant functions), and a frame averaging approach (which involves mapping each point to a set of orbit representatives). We illustrate the constructions theoretically and empirically on the unit-rescaling group and the Lorentz group. Our results show that in the non-compact case, the Weyl trick and frame averaging do not provide a consistent improvement over the baseline, except where the test data is out-of-distribution (an orders-of-magnitude improvement in this case) and, for the Weyl trick, in low-data regimes. The orthogonal projection can provide improvements in-distribution and out-of-distribution. We also observe that models trained directly on invariant features matched or beat the projection of an unconstrained one. We conclude that post-hoc symmetrization extends to non-compact groups, but it does not consistently outperform training a symmetric model.
Figures & tables
Figure 1: Example comparisons of time evolutions of the concentration of two datasets. ( 1(a) ) After t=10 steps we visually start to observe the benefits of group averaging. The equivariant model has a lower loss than the baseline at all times and is closer to the simulation (truth). At later times significant deviations between the approaches become apparent. ( 1(b) ) The equivariant model captures the features of the simulation relatively well for long times, while the baseline model develops several fictitious clusters.
Table 1: Average loss values for the datasets. The different starting positions in the trajectory and the used groups are given. The loss values are the averages of the LVRMSE over all trajectories and state variables in the respective test sets. The loss is evaluated at the time steps indicated above each column. The rollout loss is the sum of the first 15 errors after the initial sequence. The continuous symmetries S1 and T2 are approximated with n=1 random group elements, except in ( 1(d) ), as indicated.
Task
Training
Model
Fixed units
Transformed units (small shift)
Transformed units (large shift)
Regression
Fixed units
Baseline
0.0164±0.00647
(1.42±4.27)×1013
(7.12±21.9)×1026
(NRMSE)
Weyl-Projected
0.0806±0.0641
0.0806±0.0637
0.0814±0.0656
Empirical L2(μ)
0.00936±0.00474
0.0100±0.00413
0.0101±0.00379
Invariant regression
0.00487±0.00125
0.00532±0.00157
0.00525±0.00163
Frame average m=4
2.11±1.26
2.02±1.14
2.31±1.48
Transformed
Baseline
0.0325±0.0152
0.703±1.96
(1.20±3.55)×1013
Table 2: Regression uses NRMSE and classification uses expected Bernoulli KL; lower is better. Here, training a new invariant model gives the best result in all cases.
Figure 3: Test MSE for the baseline polynomial, its exact Weyl projection, the finite spatial O(3) average, and the Gram-feature invariant baseline. Both axes are logarithmic.
Figure 4: The averaging procedures on average decrease the the rollout loss. We compare 3 different averages to the baseline prediction (the model evaluated only once). On the right, we see with an example how group averaging improves prediction: The averaged trajectory (red) stay closer to the truth (black) for longer.
Model
1
10
100
Rollout
Baseline
0.0018
0.0126
0.2321
0.1497
Boosts
0.0014
0.0100
0.1911
0.1201
Inversions
0.0010
0.0065
0.1298
0.0786
Extended frame
0.0003
0.0027
0.0674
0.0313
Table 3: Average invariant loss values d for the test-set ( N=300 test trajectories) after n=1,10,100 autoregressive steps with Δτs=0.1 per step. The rollout loss is the sum of the first 15 errors.
Predictor
Accuracy
N−+
N+−
ΔNcorrect
Rej50%
Rej30%
Baseline (ParticleNet-Lite)
0.825031±0.000293
—
—
—
33.294±0.229
84.064±0.820
Collider-aligned frame average ( K=4 )
0.825078±0.000308
2026.0±38.7
2016.6±48.2
+9.4±36.7
33.408±0.218
84.597±0.503
SO(2) average
0.825032±0.000295
0.33±0.50
0.22±0.44
+0.11±0.78
33.294±0.229
84.064±0.820
SO(3) average
0.799649±0.001650
5804.3±121.5
10880.8±408.2
−5076.4±313.6
28.964±0.252
74.424±1.040
Longitudinal boost average
0.825184±0.000330
287.7±16.0
257.1±12.9
+30.6±22.8
33.350±0.200
84.007±0.679
Transverse boost average
0.824858±0.000412
951.3±47.5
986.0±46.1
−34.7±30.5
33.259±0.177
83.587±0.776
Table 4: ParticleNet-Lite performance before and after spatial-rotation, bounded-boost, and input-dependent collider-aligned frame averaging. Entries are the mean ± sample standard deviation over nine independently trained models evaluated on the same 200,000-jet test set. The spatial-rotation and bounded-boost entries use 128 frames and average class probabilities. The collider-aligned entry uses four deterministic canonical views and averages pre-softmax binary log-odds. The boost rapidities are sampled from a rejection-truncated Gaussian ρ∼N(0,0.32) , with ∣ρ∣≤0.3 . Here N−+ counts incorrect-to-correct changes and N+− counts correct-to-incorrect changes.
Figure 5: ROC AUC and background rejection at quark efficiencies of 30% and 50% for the degree-three polynomial classifier.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Table 5: Dataset specifications
Generator
Pythia 8.226, s=14TeV
Processes
qg→Z(→ννˉ)q (quark), qqˉ→Z(→ννˉ)g (gluon)
Clustering
anti- kT , R=0.4 ( FastJet 3.3.0)
Selection
pTjet∈[500,550]GeV , ∣yjet∣<1.7
Constituent features
(pT,y,ϕ,PDGid)
Split (train/val/test)
1.6×106/2×105/2×105
Classes
quark (signal) vs. gluon (background), balanced
Appendix
Table 6: Specifications of the quark–gluon jet dataset [ 52 ] .
Operator
Mean error
Maximum error
Deterministic
7.32×10−16
1.11×10−15
M=32
0.0681
0.449
M=128
0.0325
0.203
M=256
0.0206
0.109
Appendix
Table 7: Relative prediction error against exact coefficient filtering. Monte Carlo rows aggregate combinations of task, training regime, seed, and independent phase sequence.
Training
N
Model, W=500
Fixed units
Transformed units (small shift)
Transformed units (large shift)
Fixed units
50
Baseline
1.03±0.594
(3.83±7.52)×1011
(1.08±2.21)×1028
Weyl-Projected
0.776±0.0616
0.770±0.0643
0.776±0.0602
Empirical L2(μ)
0.662±0.499
0.629±0.453
0.753±0.741
Invariant regression
0.0224±0.0215
0.0222±0.0204
0.0229±0.0211
Frame average m=4
11.7±9.38
13.5±13.6
12.8±10.5
3000
Baseline
0.0167±0.00563
(5.01±14.0)×1011
(1.74±4.17)×1027
Appendix
Table 8: Test NRMSE at W=500 for two training-set sizes. Rows give the training regime and the model; columns give the unit system of the test data. Entries are the mean ± sample standard deviation over seeds.
Training
W
Model, N=3000
Fixed units
Transformed units (small shift)
Transformed units (large shift)
Fixed units
25
Baseline
0.00532±0.00281
(8.17±18.1)×105
(3.24±6.23)×1017
Weyl-Projected
0.00506±0.00157
0.00488±0.00146
0.00519±0.00170
Empirical L2(μ)
0.00452±0.00228
0.00425±0.00176
0.00449±0.00197
Invariant regression
0.00435±0.00185
0.00422±0.00151
0.00436±0.00166
Frame average m=4
0.281±0.182
0.266±0.210
0.310±0.236
250
Baseline
0.0154±0.00737
(1.74±3.71)×1011
(2.16±5.64)×1026
Appendix
Table 9: Test NRMSE at N=3000 for two non-invariant feature counts. Rows give the training regime and the model; columns give the unit system of the test data. Entries are the mean ± sample standard deviation over ten seeds.
Harvard John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA 02138, USA. · Technical University of Munich (CIT, MCML, MDSI) · MIT Computer Science and Artificial Intelligence Laboratory (CSAIL)