Authors: Henning Bahl, Victor Bresó-Pla, Anja Butter, Joaquín Iturriza Ramirez
Organizations: Institut für Theoretische Physik, Universität Heidelberg, Germany · Department of Physics, Harvard University, 02138 Cambridge, MA, USA · LPNHE, Sorbonne Université, Université Paris Cité, CNRS/IN2P3, Paris, France
Scaling laws describing the dependence of neural network performance on the amount of training data, the spent compute, and the network size have emerged across a huge variety of machine learning task and datasets. In this work, we systematically investigate these scaling laws in the context of amplitude surrogates for particle physics. We show that the scaling coefficients are connected to the number of external particles of the process. Our results demonstrate that scaling laws are a useful tool to achieve desired precision targets.
Figures & tables
abbreviation
MLP-I
LLoCa-Transformer
Network size
N
[102,106]
6×106
Dataset size
Dtrain
[10,106]
[10,106]
FLOPs
C
[1010,1017]
[1010,1017]
Table 1: Ranges for the number of network parameters, data points, and compute in FLOPs considered for our scaling law tests.
Figure 1: Left: MSE test error for the qqˉ→ttˉH amplitude using the MLP-I model as a function of the initial learning rate for different network sizes. Right: Same as left, but the test error is shown as a function of the compute in FLOPs for different training dataset sizes. The fitted power-law functions are depicted as shaded lines.
Figure 2: Left: MSE test error for the qqˉ→ttˉH amplitude using the MLP-I model as a function of the training dataset size for different NN sizes. The fitted power-law functions are depicted as shaded lines. Right: Same as left, but the test error is shown as a function of the NN size for different training dataset sizes.
param. X
2nd param.
Xc
αX
KX
C [FLOPs]
Dtrain = 102
(2.58−0.18+0.64)×1011
3.67±0.43
(3.47−0.21+0.54)×10−1
Dtrain = 104
(7.85−0.32+0.55)×1011
2.519±0.073
(50.06−3.5+0.11)×10−4
Dtrain = 8.3×105
(6.66−0.31+0.59)×1011
2.08±0.032
(19.81−1.8+0.21)×10−8
Dtrain
NMLP = 102
(9.97−0.44+0.78)×101
1.725±0.052
(7.85−0.57+2.1)×10−4
NMLP = 104
(6.51−0.29+0.51)×101
1.811±0.027
≃0
NMLP = 105
(5.93−0.26+0.45)×101
1.893±0.027
≃0
Table 2: Scaling parameters for qqˉ→ttˉH for MLP-I trained with MSE loss.
Figure 3: Left: Intrinsic dimension extracted by the qqˉ→ttˉH surrogate as a function of the training dataset size for different NN sizes. The color bounds indicate the associated uncertainty. The intrinsic dimension is calculated based on a subset of the final-layer activations with the uncertainties corresponding to the standard deviation obtained by sampling different subsets. Right: Same as left, but the intrinsic dimension is shown as a function of the NN size for different training dataset sizes.
Figure 4: Left: MSE test error for the qqˉ→ttˉH amplitude using the MLP-I model trained with a heteroscedastic loss as a function of the compute spent for training in FLOPs for different training dataset sizes. The fitted power-law functions are depicted as shaded lines. Right: Same as left, but the test error is shown as a function of the training dataset size for different NN sizes.
Figure 5: Left: Distribution of learned uncertainty values for the qqˉ→ttˉH surrogate for different amounts of spent compute. The dataset size is fixed to Dtrain=3.16×105 . Right: Same as left, but the distributions are shown for different sizes of the training dataset. The compute is fixed to C=2.12×1015 FLOPs.
Figure 6: Left: Pull distributions for the qqˉ→ttˉH surrogate for different amounts of spent compute. Right: Same as left, but the pull distributions are shown for different sizes of the training dataset.
Figure 7: Comparison of the MSE test error for the qqˉ→ttˉH amplitude as a function of the training dataset size between the LLoCa-Transformer and the MLP-I model.
Figure 8: MSE test error as a function of the compute spent for training in FLOPs (left) and the training dataset size (right) for the qqˉ→Z+ng (upper row), gg→γγg,γγgg (middle row), and qqˉ→WZ,WZg,WWZ,WZgg amplitudes (lower row).
Figure 9: MSE test errors for the qqˉ→Zgggg amplitude comparing the MLP-I and LLoCa-Transformer models as a function of the size of the training dataset.
Figure 10: Left: Power law exponent αX as a function of the number of particles in the final state of each process for the scaling on computing resources. Right: Same as left for the scaling on the dataset size.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 11: Left: Relative L1 error for the MLP-I model as a function of the training dataset size for different numbers of total parameters of the network. Right: Same as left but using the error metric studied in Ref. [ 48 ] .
Figure 12: Left: MSE test errors for the qqˉ→Zg amplitude using the MLP-I model as a function of the size of the training dataset. Results are shown for three different numbers of trainable parameters. Right: Same as left, but for the qqˉ→Zgggg amplitude.
Figure 13: Left: MSE test errors for the qqˉ→Zg amplitude using the MLP-I model as a function of the compute spent for training in FLOPs for three different sizes of the training dataset. Right: Same as left, but for the qqˉ→Zgggg amplitude.
Figure 14: Left: MSE test errors for the gg→γγ+g amplitude using the MLP-I model as a function of the size of the training dataset. Results are shown for three different numbers of trainable parameters. Right: Same as left, but for the gg→γγ+gg amplitude.
Figure 15: Left: MSE test errors for the gg→γγ+g amplitude using the MLP-I model as a function of compute spent for training. Results are shown for three sizes of the training dataset. Right: Same as left, but for the gg→γγ+gg amplitude.
Figure 16: Upper left: MSE test errors for the qqˉ→WZ amplitude using the MLP-I model as a function of the size of the training dataset for different NN sizes. Results are shown for three different numbers of trainable parameters. Upper right: Same as upper left but for the qqˉ→WWZ amplitude. Lower left: Same as upper left but for the qqˉ→WZg amplitude. Lower right: Same as lower left but for the qqˉ→WZgg amplitude.
Figure 17: Upper left: MSE test errors for the qqˉ→WZ amplitude using the MLP-I model as a function of the compute spent for training for different sizes of the training dataset. Upper right: Same as left but for the qqˉ→WWZ amplitude. Lower left: Same as upper left but for the qqˉ→WZg amplitude. Lower right: Same as lower left but for the qqˉ→WZgg amplitude.
Figure 18: Intrinsic dimension extracted by the qqˉ→Z+gluons surrogates as a function of the training dataset size for different NN sizes. The color bounds indicate the associated uncertainty. The intrinsic dimension is calculated based on a subset of the final-layer activations with the uncertainties corresponding to the standard deviation obtained by sampling different subsets.
Figure 19: Intrinsic dimension extracted by the gg→γγg,γγgg surrogates as a function of the training dataset size for different NN sizes. The color bounds indicate the associated uncertainty. The intrinsic dimension is calculated based on a subset of the final-layer activations with the uncertainties corresponding to the standard deviation obtained by sampling different subsets.
Figure 20: Intrinsic dimension extracted by the qqˉ→WZ,WWZ,WZg surrogates as a function of the training dataset size for different NN sizes. The color bounds indicate the associated uncertainty. The intrinsic dimension is calculated based on a subset of the final-layer activations with the uncertainties corresponding to the standard deviation obtained by sampling different subsets.
param. X
2nd param.
Xc
αX
KX
Dtrain
NMLP = 104
(6.98−0.31+0.56)×101
1.752±0.028
(2.23−0.22+387)×10−7
Dtrain
NMLP = 105
(2.55−0.12+0.23)×101
1.949±0.030
(4.86−0.46+7.3)×10−8
Dtrain
NMLP = 106
(2.073−0.096+0.18)×101
2.239±0.051
(3.10−0.22+0.76)×10−6
C [FLOPs]
Dtrain = 102
(6.11−0.35+0.80)×1010
1.89±0.94
(3.11−0.19+0.48)×10−2
C [FLOPs]
Dtrain = 104
(12.43−0.49+0.81)×1010
2.13±0.037
(2.21−0.16+0.54)×10−6
C [FLOPs]
Dtrain = 106
(11.95−0.48+0.81)×1010
2.017±0.027
(4.92−0.39+1.9)×10−8
Appendix
Table 3: Fitted scaling parameters for ttˉH MLP-I surrogates trained with heteroscedastic loss.
param. X
2nd param.
Xc
αX
KX
Dtrain
NLLoCa-Tr = 6×106
6.09−0.29+0.54
1.303±0.015
≃0
Appendix
Table 4: Fitted scaling parameters for ttˉH LLoCa-Transformer surrogates trained with MSE loss.
param. X
2nd param.
Xc
αX
KX
Dtrain
NMLP = 104
8.55−0.45+0.96
1.958±0.028
(1.31−0.13+14)×10−9
NMLP = 105
8.73−0.44+0.89
2.084±0.027
≃0
NMLP = 106
10.1−0.48+0.91
2.243±0.027
(1.61−0.15+2.4)×10−10
C [FLOPs]
Dtrain = 102
(7.38−0.54+2.0)×1010
1.84±0.18
(5.09−0.33+0.91)×10−3
Dtrain = 104
(2.28−0.10+0.19)×1011
3.129±0.070
(7.21−0.50+1.6)×10−8
Dtrain = 3.03×105
(20.97−0.94+1.7)×1010
3.039±0.050
(8.88−0.66+2.6)×10−10
Appendix
Table 5: Fitted scaling parameters for Zg MLP-I surrogates trained with MSE loss.
param. X
2nd param.
Xc
αX
KX
Dtrain
NMLP = 102
(2.65−0.16+0.43)×101
1.373±0.055
(6.81−0.50+1.9)×10−4
NMLP = 104
15.21−0.97+2.7
1.316±0.029
≃0
NMLP = 105
10.92−0.74+2.3
1.301±0.031
(3.24−0.32+11)×10−6
NMLP = 106
8.68−0.60+2.0
1.280±0.031
(2.50−0.25+12)×10−6
C [FLOPs]
Dtrain = 102
(1.65−0.14+1.1)×1010
1.25±0.21
(4.70−0.29+0.76)×10−2
Dtrain = 104
(3.60−0.26+0.98)×1010
1.600±0.074
(12.66−0.85+2.6)×10−5
Appendix
Table 6: Fitted scaling parameters for Zgg MLP-I surrogates trained with MSE loss.
param. X
2nd param.
Xc
αX
KX
Dtrain
NMLP = 102
(5.00−0.31+0.81)×101
1.193±0.062
(6.68−0.48+1.8)×10−3
NMLP = 104
(2.32−0.17+0.57)×101
1.030±0.030
≃0
NMLP = 105
(1.59−0.12+0.46)×101
1.008±0.030
≃0
NMLP = 106
(1.268−0.096+0.40)×101
0.997±0.031
≃0
C [FLOPs]
Dtrain = 102
(1.34−0.12+1.0)×1010
1.70±0.56
(10.58−0.63+1.5)×10−2
Dtrain = 104
(2.93−0.23+1.1)×1010
1.65±0.13
(1.68−0.11+0.28)×10−3
Appendix
Table 7: Fitted scaling parameters for Zggg MLP-I surrogates trained with MSE loss.
param. X
2nd param.
Xc
αX
KX
Dtrain
NMLP = 104
(1.98−0.16+0.85)×101
0.788±0.031
≃0
NMLP = 105
(1.39−0.12+0.72)×101
0.774±0.032
≃0
NMLP = 106
(1.010−0.087+0.62)×101
0.765±0.034
≃0
C [FLOPs]
Dtrain = 102
(1.23−0.12+7.0)×1010
1.86±0.16
(2.39−0.14+0.34)×10−1
Dtrain = 104
(3.07−0.26+1.6)×1010
1.45±0.14
(5.09−0.32+0.85)×10−3
Dtrain = 3.13×105
(1.68−0.15+1.5)×1010
1.002±0.055
(3.79−0.28+1.1)×10−4
Appendix
Table 8: Fitted scaling parameters for Zgggg MLP-I surrogates trained with MSE loss.
param. X
2nd param.
Xc
αX
KX
Dtrain
NLLoCa-Tr = 6×106
3.76−0.20+0.44
0.911±0.013
≃0
Appendix
Table 9: Fitted scaling parameters for Zgggg LLoCa-Transformer surrogates trained with MSE loss.
param. X
2nd param.
Xc
αX
KX
Dtrain
NMLP = 104
3.71−0.24+0.69
1.588±0.028
≃0
NMLP = 105
5.77−0.34+0.84
1.738±0.027
≃0
NMLP = 106
5.08−0.31+0.77
1.720±0.027
≃0
C [FLOPs]
Dtrain = 102
(5.12−0.41+2.1)×1010
1.88±0.20
(3.59−0.23+0.63)×10−3
Dtrain = 104
(7.50−0.57+2.3)×1010
1.565±0.049
(2.02−0.16+0.76)×10−6
Dtrain = 3.86×105
(11.58−0.72+1.9)×1010
1.812±0.030
(2.02−0.20+11)×10−9
Appendix
Table 10: Fitted scaling parameters for WZ MLP-I surrogates trained with MSE loss.
param. X
2nd param.
Xc
αX
KX
Dtrain
NMLP = 104
14.58−0.99+3.0
1.223±0.029
≃0
NMLP = 105
(1.63−0.10+0.29)×101
1.310±0.028
≃0
NMLP = 106
13.21−0.88+2.7
1.262±0.029
≃0
C [FLOPs]
Dtrain = 102
(6.35−0.53+3.1)×1010
1.98±0.59
(7.14−0.44+1.2)×10−2
Dtrain = 104
(10.69−0.87+4.6)×1010
1.506±0.088
(2.89−0.20+0.67)×10−4
Dtrain = 3.82×105
(7.74−0.60+2.7)×1010
1.307±0.036
(1.39−0.13+1.61)×10−6
Appendix
Table 11: Fitted scaling parameters for WZg MLP-I surrogates trained with MSE loss.
param. X
2nd param.
Xc
αX
KX
Dtrain
NMLP = 102
(4.72−0.30+0.81)×101
1.154±0.058
(5.28−0.39+1.51)×10−3
NMLP = 104
(2.67−0.19+0.65)×101
0.995±0.030
≃0
NMLP = 105
(1.83−0.14+0.56)×101
0.959±0.034
≃0
NMLP = 106
(1.36−0.12+0.78)×101
0.932±0.037
≃0
C [FLOPs]
Dtrain = 102
(1.12−0.11+7.0)×1010
1.61±1.0
(2.12−0.13+0.31)×10−1
Dtrain = 104
(2.66−0.23+1.5)×1010
1.39±0.12
(2.39−0.15+0.42)×10−3
Appendix
Table 12: Fitted scaling parameters for WZgg MLP-I surrogates trained with MSE loss.
param. X
2nd param.
Xc
αX
KX
Dtrain
NMLP = 104
13.7−0.91+2.7
1.273±0.029
≃0
NMLP = 105
10.6−0.72+2.3
1.256±0.028
≃0
NMLP = 106
8.43−0.59+2.0
1.227±0.028
≃0
C [FLOPs]
Dtrain = 102
(5.14−0.46+4.5)×1010
1.79±0.48
(4.99−0.31+0.82)×10−2
Dtrain = 104
(7.95−0.68+4.5)×1010
1.405±0.083
(2.29−0.16+0.56)×10−4
Dtrain = 3.85×105
(6.75−0.53+2.5)×1010
1.316±0.037
(1.15−0.11+1.2)×10−6
Appendix
Table 13: Fitted scaling parameters for WWZ MLP-I surrogates trained with MSE loss.
param. X
2nd param.
Xc
αX
KX
Dtrain
NMLP = 104
14.1−0.92+2.6
1.304±0.029
≃0
NMLP = 105
(1.72−0.10+0.25)×101
1.461±0.027
≃0
NMLP = 106
14.7−0.89+2.2
1.445±0.027
≃0
C [FLOPs]
Dtrain = 102
(6.17−0.57+7.4)×1010
1.21±0.28
(7.98−0.52+1.5)×10−2
Dtrain = 104
(1.74−0.12+0.43)×1011
1.681±0.079
(1.42−0.10+0.34)×10−4
Dtrain = 4.44×105
(13.6−0.93+3.0)×1010
1.516±0.033
(1.73−0.16+3.1)×10−7
Appendix
Table 14: Fitted scaling parameters for γγg MLP-I surrogates trained with MSE loss.
param. X
2nd param.
Xc
αX
KX
Dtrain
NMLP = 104
(1.49−0.12+0.50)×101
0.939±0.031
≃0
NMLP = 105
11.77−0.90+3.8
0.988±0.029
≃0
NMLP = 106
9.50−0.74+3.4
0.967±0.029
≃0
C [FLOPs]
Dtrain = 102
(1.35−0.13+2.7)×1010
1.03±0.27
(12.04−0.75+2.0)×10−2
Dtrain = 104
(4.45−0.36+1.8)×1010
1.261±0.083
(1.79−0.12+0.36)×10−3
Dtrain = 4.44×105
(3.66−0.29+1.3)×1010
1.112±0.034
(1.60−0.14+0.99)×10−5
Appendix
Table 15: Fitted scaling parameters for γγgg MLP-I surrogates trained with MSE loss.
Department of Applied Physics Stanford University Stanford, CA 94305, USA · Department of Physics and Astronomy University of California, Irvine Irvine, CA 92697, USA · Department of Particle Physics and Astrophysics SLAC National Accelerator Laboratory Stanford University Menlo Park, CA 94025, USA