hep-phOct 26, 2026
SaveThe proton structure one mode at a time: a Neural Tangent Kernel analysis of a DIS fit
Organizations: The Higgs Centre for Theoretical Physics, School of Physics and Astronomy, The University of Edinburgh, Peter Guthrie Tait Road, Edinburgh EH9 3FD, United Kingdom
Abstract
We study the training dynamics of a feed-forward neural network (NN) through the analysis of the neural tangent kernel (NTK) in a phenomenologically relevant fit of parton distribution functions (PDFs) from deep inelastic scattering (DIS) data. The exact integration of the gradient flow derived in a previous single-flavour analysis is generalised to the multi-flavour case, yielding an analytical perspective on the relationship between data and methodology during training. The Shapley value recently used in the context of PDFs is derived within our statistical framework for observables that depend linearly on the PDFs, and we outline a geometric interpretation of training. We expose the role of gradient-based optimisation algorithms and of the parametrisation in the regularisation of the ill-posed inverse problem that affects the determination of PDFs from experimental data, demonstrating that the regularisation amounts to the incremental inclusion of NTK modes, each learned at the timescale set by the corresponding eigenvalue. We find that only a few leading modes of the NTK are necessary to reconstruct the target PDFs, while the noise of the data is carried by the sub-leading ones learned at later stages of training. Finally, we consider the impact of the initialisation on the training dynamics, and recover a transient behaviour observed in the previous analysis and in the machine learning (ML) literature. Beyond sharpening our understanding of the training dynamics, our framework can be applied to other parametrisations (such as fixed functional forms) and ML applications in physics, as well as to stress tests in the presence of inconsistent data.
Figures & tables
Figure 1: The block matrices of the operator for the eight independent fitted flavours. Each block represents the correlations between a pair of flavours on the -grid as seen through the FK tables and weighted by the experimental uncertainties. The colour bar on the right encodes the size of these correlations: red for the correlated points, blue for the anti-correlated ones, white for points not constrained by the data. The operator is symmetric by construction, so that only the lower triangular block is shown.
Figure 2: Schematic representation of the training dynamics in the functional manifold . (Left) Training is a curve on (solid blue). Freezing the NTK at selects the tangent plane , on which the flow continues (dashed red). The curvature of the manifold separates the two trajectories (dotted). (Right) In this work the frozen flow starts from an independent initialisation , and lives on the affine plane parallel to the tangent plane. Their separation along is conserved by the flow.
Figure 3: The output of the ensemble of models at initialisation at the initial scale , here for , , , and . We compare the case of the model with preprocessing factors (blue) and without preprocessing factors (orange), the latter corresponding to the contribution of the neural network alone. The solid lines represent the mean value computed over an ensemble of 120 replicas, while the shaded bands represent the one-sigma uncertainty computed as the variance over the same ensemble. The 68 c.l. is also shown as dashed lines.
Figure 4: Continuation of Fig. 3 with , , , and .
Figure 5: The evolution of the loss function during training for both the training (blue) and validation (orange) sets of data. The loss is the first term of eq. 10 , normalised to the number of data points of the corresponding set. The lines represent the mean value computed over an ensemble of 120 replicas, while the shaded bands represent the one-sigma uncertainty computed as the variance over the same ensemble. Note the logarithmic scale in the horizontal axis.
Figure 6: Eigenvalues of the NTK during training for both cases without (left column) and with (right column) preprocessing factors. The figure displays the first 200 ranks of the NTK spectrum, distributed evenly across the four rows. The colour of each curve encodes the rank of the eigenvalues, as indicated by the colour bar. The lines represent the median value computed over an ensemble of 120 replicas. The dashed black line marks the separation between the image and the kernel of the NTK, i.e. the numerical-zero relative threshold , with . The dimension of Fig. 8 is precisely the number of eigenvalues lying above that line at each training time.
Figure 7: Close-up view of the leading ten eigenvalues of the NTK during training for both cases without (left column) and with (right column) preprocessing factors. As in Fig. 6 , the colour of each curve encodes the rank of the eigenvalues, as indicated by the colour bar. Here the solid lines represent the median value while the shaded bands represent the 68% c.l., both computed over an ensemble of 120 replicas.
Figure 8: Dimension of the image of the NTK during training as a function of the training time. We show both the case of the full model with preprocessing factors (blue) and the case of the NN alone (orange). The solid line represents the mean value computed over an ensemble of 120 replicas, while the shaded bands represent the one-sigma uncertainty computed as the variance over the same ensemble. At each training time, this dimension is the number of eigenvalues lying above the image/kernel threshold drawn in Fig. 6 .
Figure 9: The modes of the training Hamiltonian, eq. 27 , as functions of and shown at three training epochs: (top block), (middle block), and (bottom block). The eight panels of each block show, for each fitted flavour, the ranks coloured as indicated in the legend. The modes are those of a single replica. The vertical scale differs between flavours but is the same across the three epochs. The overall sign of each mode is chosen so that its coefficient in the expansion of on the modes is positive. The dashed black line is the central replica of the underlying PDF used to generate the data, rescaled for each flavour by the factor indicated, common to the three epochs.
Figure 10: Characteristic resolution times , in epochs, of every mode in at the frozen epoch (ordered by decreasing ). The threshold separating the image from the kernel is obtained with , as in Fig. 8 , giving median modes at this epoch. The red dashed line marks the training horizon ( epochs). The dotted lines are the three evolution times of Fig. 11 . Points are medians over the ensemble of replicas.
Figure 11: The analytical solution (blue) of the eight fitted flavours (rows) computed with the NTK frozen at and for five evolution times (columns). The initial condition of the analytical solution is an independent ensemble of PDFs at initialisation for all panels. In the first three columns the analytical solution is compared against the trained network (orange) evaluated at the same training time, while in the last two columns the trained network is kept fixed at the last available epoch, . Curves show the mean, one-sigma band and c.l. over the ensemble of replicas. The underlying law is also drawn as a dashed black line.
Figure 12: Analytical (blue) versus empirically trained (orange) evolution of the singlet , for three choices of the epoch at which the NTK is frozen ( , rows) and the same three evolution times as in Fig. 11 (columns). Statistics and the underlying law (dashed black) are as in Fig. 11 .
Figure 13: Decomposition of the analytical solution of eq. 30 into the contribution from the initial condition, (green), and the contribution from the data, (purple, hatched), together with their sum (blue), for the eight fitted flavours (rows) and the five evolution times of Fig. 11 (columns). The NTK is frozen at epoch . The initial condition and the statistics over the ensemble of replicas are as in Fig. 11 . The dashed black line is the underlying law .
Figure 14: Fractions of the variance of the analytical solution carried by the initial condition (green), by the data (purple), and by their correlation (dotted grey). Each curve is the ratio between the diagonal entries of the corresponding terms in eq. 67 and the total covariance, so that all fractions sum to one at each point in . The layout and settings are the same as in Fig. 11 . The two horizontal grey lines mark zero and one of the y-scale. The point at , where PDFs vanish by construction, is omitted.
Figure 15: One-sigma width of the PDFs at initialisation as a function of at the initial scale , computed as the standard deviation over replicas of the full model, preprocessing factors included. Each panel shows one of the eight fitted flavours, and each curve one of the six values of the initialisation exponent of eq. 68 . Both axes are logarithmic.
Figure 16: Ratio of the widths of Fig. 15 for consecutive values of the initialisation exponent, one panel per fitted flavour. The horizontal dotted line is the value predicted in the linear regime by eq. 69 . The ratio is close to this value for , whereas for smaller it falls below it, most strongly at small , where the input is largest and the hyperbolic tangents saturate.
Figure 17: Training (solid) and validation (dashed) losses as functions of the epoch, for a single representative replica. Each curve represents one of the six values of considered in this section. Both axes are logarithmic. For the trajectory develops a plateau, on which the loss stalls close to its initial value, followed by a rapid drop. Increasing delays that drop and slightly lifts the level reached afterwards.
Figure 18: Training trajectories of the leading eigenvalues of the NTK, with preprocessing factors excluded. Each panel corresponds to a different initialisation exponent of eq. 68 . The colour encodes the eigenvalue rank , as in Fig. 6 . The curves are medians over the replicas of each fit, and bands are omitted for clarity; the red dashed line marks the median escape epoch of each fit. The abscissa is linear below epochs and logarithmic above, so that the initialisation at fits on the axis. The ordinate is cut at , the smallest eigenvalue that still lies inside at the tolerance of Fig. 8 for any of these fits.
Figure 19: Displacements of the parameters in Euclidean space as a function of training time and for the different initialisation widths. (Left) The distance separating the parameters from their position at initialisation. (Right) The same distance normalised to the size of the initialisation, . Only the network parameters are shown, the preprocessing factors being frozen after initialisation. Solid lines are means over the replicas of each fit and the shaded areas are one-sigma bands. The dotted vertical lines mark the median escape epoch of each fit. The abscissa is linear below epochs and logarithmic above, so that the initialisation at sits on the axis. The first stored epoch is omitted, the displacement vanishing there by construction.
Figure 20: Training loss of the analytical solution of eq. 30 with the NTK frozen at initialisation (dashed) against the loss of the empirically trained network (solid), one panel per initialisation exponent . The ordinate is logarithmic while the abscissa is linear below epoch and logarithmic above. The analytical loss is evaluated by feeding the evolved solution back through the same FK tables and inverse covariance that define the trained loss, so that the two curves are the same observable. Both are medians over the replicas of each fit. The dotted vertical line marks the median escape epoch of the ensemble. The trained curves are those of Fig. 17 , up to the replacement of the single representative replica by the ensemble median.
Figure 21: Analytical (blue) versus empirically trained (orange) solution of the singlet , with the NTK frozen at initialisation, . The rows show the analytical solution obtained with (top), (middle), (bottom). The columns represent three different evolution epochs common to the three rows. Analytical and trained solutions are compared at the same epoch . The solid line represents the mean over the ensemble, the shaded area the one-sigma band and the dashed lines the c.l. over the replicas of each fit. The dashed black line is the underlying law used to generate the data.
Figure 22: Same as Fig. 20 , but with the NTK frozen at the last stored epoch, .
Figure 23: Same as Fig. 21 , but with the NTK frozen at the last stored epoch, .
Figure 24: (Left) Minimum over the first epochs of the validation loss, minus that of for the same replica: mean over replicas with its bootstrap error (thick bar), and interval of the replicas (thin line). The ordinate is linear within and logarithmic outside. (Middle) Number of epochs for the training loss to fall below , divided by that of for the same replica: median with its bootstrap interval (thick bar), and interval of the replicas (thin line); the arrow marks , where of the replicas never reach within epochs. (Right) , with the largest eigenvalue of , at initialisation (open) and at the end of the training (filled), median and interval over five replicas. The dotted line represents the stability condition of GD, eq. 73 . We use black circles for the fixed learning rate, , and orange squares for the rescaled learning rates so that at initialisation matches that of ( for and for ), drawn slightly to the right.
Figure 25: Comparison between the trained solution with the contribution of the data removed, (black), and the contribution of the initial condition to the analytical solution, (green), for the eight fitted flavours (rows) and for , , , and (columns). For each fit the NTK is frozen at the last epoch, , and both quantities are evaluated at the end of the training, with corresponding to . The two coincide only if the analytical solution reproduces the trained network. The solid lines represent the mean over the ensemble, the shaded areas the one-sigma band and the dashed lines the c.l. over the replicas of each fit.
Figure 26: Eigenvalues of the NTK of the neural network (preprocessing factors excluded) at initialisation, against their rank. Coloured curves are the baseline architecture at . The black curve is the wide architecture at . Lines are medians over the replicas of each fit ( for the baseline architecture, for the wide one), bands are intervals. Each curve stops at the dimension of the image of the NTK, with the tolerance of Fig. 8 .
Figure 27: Training loss at for the baseline architecture (learning rate ) and the wide architecture ( ), as a function of the epoch (left) and of the gradient-flow time (right). Lines are medians and bands intervals over the first five replicas of each fit. Both axes are logarithmic.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 28: Saturation of the first (left) and second (right) hidden layers as a function of as defined in eq. 82 . The six values of considered in this analysis are represented by the different coloured curves. The curves are means over the replicas of each fit. For the saturation is negligible across the entire range, whereas for smaller values of the saturation becomes significant, particularly at small where the input is largest.
Figure 29: The spectrum of against the eigenvalue rank at initialisation, for the six initialisation exponents and computed from the neural network alone (no preprocessing). (Left) The eigenvalues normalised to the leading one, so that the six fits share an ordinate. (Right) The gap between consecutive eigenvalues, , taken replica by replica before the median. Curves are medians over the replicas and bands are intervals; there is one marker per eigenvalue. The dotted vertical lines mark the three tier boundaries , and predicted by eq. 71 . Note that the curve for each stops at its own of eq. 149 , that is, the last point before .
Figure 30: (Left) The dimension of the image of the NTK, , as a function of the training epoch, for the six values of considered in this analysis. The tolerance is set as in Fig. 8 . Preprocessing factors are excluded from the NTK. The curve with is the same as Fig. 8 without preprocessing factors. (Right) The saturation of eq. 82 , averaged over the input grid in , as a function of the training epoch. Here shown for the first (solid) and second (dashed) hidden layer, whose values at are those tabulated in eq. 83 . In both panels, solid lines are medians over the replicas of each fit and the shaded bands are the intervals; the vertical dotted lines mark the median escape epoch of the corresponding fit. The abscissa is linear below epochs and logarithmic above, so that the initialisation at fits on the axis.
Figure 31: The singlet (upper block) and gluon (lower block) components of six modes of the training Hamiltonian, eq. 27 , as functions of , for the six values of the initialisation exponent . Within each block the three rows are the epoch at which the mode is evaluated: initialisation, the escape epoch of each fit, and the last stored epoch , well beyond the drop of the loss for every . The columns are the mode rank, taken as the first and the last of each tier of . The modes are computed for the bare network, with the preprocessing factors removed, on a single replica common to the six fits. Each curve is normalised by its own signed value at the point of largest , independently for each flavour and each rank, so that the largest excursion of every curve is . Note that in logarithmic scale a straight line is a pure dependence.
Figure 32: Residual of eq. 150 as a function of the rank , at initialisation (left) and at the escape epoch (right). Curves are medians over the replicas of each fit and bands are intervals. The curves for and terminate at the median of the corresponding fit ( cfr. Fig. 30 ). The dotted vertical lines mark the boundaries of the tiers as in Fig. 29 . The wide band on the curve of the right panel is due to the replicas out of that never escape within the training and therefore remain close to .
Figure 33: Subspace drift of eq. 151 , measuring how far the leading eigenvectors of have rotated away from the subspace they spanned at initialisation, as a function of the training epoch. One panel per subspace size , sharing both axes. The first five panels use a value of common to all six fits, chosen at the tier boundaries , and together with and within the first tier. The last panel sets to the dimension of the image at initialisation, . Curves are medians over the replicas of each fit and bands are intervals; the dotted vertical lines mark the median escape epoch . The NTK is that of Fig. 18 , with the neural network alone. The initialisation is omitted, vanishing there by construction.
Figure 34: Characteristic resolution times of eq. 66 , in epochs, of every mode in with the NTK frozen at initialisation, against the mode rank and for the six initialisation exponents . The kernel is that of the full model, as in Fig. 10 , so that each curve terminates at the corresponding entry of eq. 70 . Points are medians over the replicas of each fit. The red dashed line marks the training horizon epochs, the last epoch of the numerical training: a mode whose resolution time lies above it has not had the time to be learned. The dotted vertical line marks the modes of the constant sector. The star on each curve is the slowest mode that the corresponding fit could have resolved by its own escape epoch .
Figure 35: The discrepancy of eq. 153 between the analytical and the trained solution at the end of the training, as a function of the epoch at which the NTK is frozen, for the six values of the initialisation exponent . (Left) Against in epochs, with the median escape epoch of each fit marked by a dotted vertical line of the same colour. The abscissa is logarithmic, so that the point at is shown only in the right panel. (Right) The same curves against in units of the escape epoch of each fit, on a linear abscissa truncated at . The dashed grey line marks . Markers are the sampled freezing epochs, and lines only interpolate between them. is computed from the ensemble means over the replicas of each fit.
More in this topic
Neural Network Training Dynamics
Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.
See all 516 papers