Organizations: MIPT, Dolgoprudny, 141701, Russia · NRC “Kurchatov Institute", 123182, Moscow, Russia · Institute for Information Transmission Problems, Moscow 127994, Russia · ITEP, B.Cheremushkinskaya, 25, 177259, Moscow, Russia
Many scientific problems in the context of machine learning can be reduced to the search of polynomial answers in appropriate variables. The Hevisidization of arbitrary polynomial is actually provided by one-and-the same two-layer expression. What prevents the use of this simple idea is the fatal degeneracy of the Heaviside and sigmoid expansions, which traps the steepest-descent evolution at the bottom of canyons, close to the starting point, but far from the desired true minimum. This problem is unavoidable and can be formulated as a peculiar uncertainty principle -- the sharper the minimum, the smoother the canyons. It is a direct analogue of the usual one, which is the pertinent property of the more familiar Fourier expansion. Standard machine learning software fights with this problem empirically, for example, by testing evolutions, originated at randomly distributed starting points and then selecting the best one. Surprisingly or not, phenomena and problems, encountered in ML application to science are pure scientific and belong to physics, not to computer science. On the other hand, they sound slightly different and shed new light on the well-known phenomena -- for example, extend the uncertainty principle from Fourier and, later, wavelet analysis to a new peculiar class of nearly singular sigmoid functions.
Figures & tables
Figure 1: Two different stable network parameters states (shown are about 150 ordered values), as the results of training for the same dataset but with different initial values
Figure 2: Matching of 400 training target determinant values and a trained network output
Figure 3: Mismatch of the output returned by the same network, but taken for another set of 400 testing values
Figure 4: The minimal curve γ for L(a,b;1,1)
Figure 5: The look of the plot of L(a,b) in the region a∈[−0.5,3.5],b∈[−0.5,3.5] for K=12 .
Figure 6: σ(x)−x for α=0.2,1.0,1.1
Figure 7: dependency on the sign of ξ
Figure 8: dependency on ϵ=1,0.5,0.4
Figure 9: Training with 10 samples. Left: values of b0,w0,b2,w2 , middle: values of w1 (close to diagonal), right: matching training sample (orange y(a) completely overlap blue original values of a )
Figure 10: Training with 40 samples. Left: values of b0,w0,b2,w2 (practically the same as for 10 samples), middle: values of w1 (essentially moved away from diagonal), right: matching training sample (orange y(a) completely overlap blue original values of a )
Figure 11: Evolution of 3×3 -determinant model output (top) and some learnable parameters (bottom) after 2000 and 6000 training steps correspondingly: the case of initializing parameters according to determinant ansatz - essential decreasing loss values, minor corrections of network weights
Figure 12: Evolution of 3×3 -determinant model output (top) and some learnable parameters (bottom) after 2000 and 6000 training steps correspondingly: the case of randomly initialized parameters - high remaining loss value, active changes of network weights
Department of Industrial Engineering of the Università di Trento, Trento, Italy · Department of Philosophy of the Università Cattolica di Milano, Milano, Italy