hep-phOct 26, 2026

The proton structure one mode at a time: a Neural Tangent Kernel analysis of a DIS fit

Authors: Amedeo Chiefa, Luigi Del Debbio, Richard Kenway

Organizations: The Higgs Centre for Theoretical Physics, School of Physics and Astronomy, The University of Edinburgh, Peter Guthrie Tait Road, Edinburgh EH9 3FD, United Kingdom

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We study the training dynamics of a feed-forward neural network (NN) through the analysis of the neural tangent kernel (NTK) in a phenomenologically relevant fit of parton distribution functions (PDFs) from deep inelastic scattering (DIS) data. The exact integration of the gradient flow derived in a previous single-flavour analysis is generalised to the multi-flavour case, yielding an analytical perspective on the relationship between data and methodology during training. The Shapley value recently used in the context of PDFs is derived within our statistical framework for observables that depend linearly on the PDFs, and we outline a geometric interpretation of training. We expose the role of gradient-based optimisation algorithms and of the parametrisation in the regularisation of the ill-posed inverse problem that affects the determination of PDFs from experimental data, demonstrating that the regularisation amounts to the incremental inclusion of NTK modes, each learned at the timescale set by the corresponding eigenvalue. We find that only a few leading modes of the NTK are necessary to reconstruct the target PDFs, while the noise of the data is carried by the sub-leading ones learned at later stages of training. Finally, we consider the impact of the initialisation on the training dynamics, and recover a transient behaviour observed in the previous analysis and in the machine learning (ML) literature. Beyond sharpening our understanding of the training dynamics, our framework can be applied to other parametrisations (such as fixed functional forms) and ML applications in physics, as well as to stress tests in the presence of inconsistent data.

Figures & tables

Table 1
Figure 1: The block matrices of the operator MM for the eight independent fitted flavours. Each block represents the correlations between a pair of flavours on the xx -grid as seen through the FK tables and weighted by the experimental uncertainties. The colour bar on the right encodes the size of these correlations: red for the correlated points, blue for the anti-correlated ones, white for points not constrained by the data. The operator MM is symmetric by construction, so that only the lower triangular block is shown.
Figure 2: Schematic representation of the training dynamics in the functional manifold Mf\mathcal{M}_{f} . (Left) Training is a curve on Mf\mathcal{M}_{f} (solid blue). Freezing the NTK at tNTKt_{\rm NTK} selects the tangent plane TftNTKMT_{f_{t_{\rm NTK}}}\mathcal{M} , on which the flow continues (dashed red). The curvature of the manifold separates the two trajectories (dotted). (Right) In this work the frozen flow starts from an independent initialisation f0f_{0} , and lives on the affine plane f0+Im(Θ)f_{0}+\mathrm{Im}(\Theta) parallel to the tangent plane. Their separation along Ker(Θ)\mathrm{Ker}(\Theta) is conserved by the flow.
Figure 3: The output of the ensemble of models at initialisation at the initial scale Q0Q_{0} , here for Σ\Sigma , gg , VV , and V3V_{3} . We compare the case of the model with preprocessing factors (blue) and without preprocessing factors (orange), the latter corresponding to the contribution of the neural network alone. The solid lines represent the mean value computed over an ensemble of 120 replicas, while the shaded bands represent the one-sigma uncertainty computed as the variance over the same ensemble. The 68 %\% c.l. is also shown as dashed lines.
Figure 4: Continuation of Fig. 3 with V8V_{8} , T3T_{3} , T8T_{8} , and T15T_{15} .
Figure 5: The evolution of the loss function during training for both the training (blue) and validation (orange) sets of data. The loss is the first term of eq. 10 , normalised to the number of data points of the corresponding set. The lines represent the mean value computed over an ensemble of 120 replicas, while the shaded bands represent the one-sigma uncertainty computed as the variance over the same ensemble. Note the logarithmic scale in the horizontal axis.
Figure 6: Eigenvalues of the NTK during training for both cases without (left column) and with (right column) preprocessing factors. The figure displays the first 200 ranks of the NTK spectrum, distributed evenly across the four rows. The colour of each curve encodes the rank ii of the eigenvalues, as indicated by the colour bar. The lines represent the median value computed over an ensemble of 120 replicas. The dashed black line marks the separation between the image and the kernel of the NTK, i.e. the numerical-zero relative threshold εtol λ1(t)\varepsilon_{\rm tol}\,\lambda_{1}(t) , with εtol=10−12\varepsilon_{\rm tol}=10^{-12} . The dimension of Fig. 8 is precisely the number of eigenvalues lying above that line at each training time.
Figure 7: Close-up view of the leading ten eigenvalues of the NTK during training for both cases without (left column) and with (right column) preprocessing factors. As in Fig. 6 , the colour of each curve encodes the rank ii of the eigenvalues, as indicated by the colour bar. Here the solid lines represent the median value while the shaded bands represent the 68% c.l., both computed over an ensemble of 120 replicas.
Figure 8: Dimension of the image of the NTK during training as a function of the training time. We show both the case of the full model with preprocessing factors (blue) and the case of the NN alone (orange). The solid line represents the mean value computed over an ensemble of 120 replicas, while the shaded bands represent the one-sigma uncertainty computed as the variance over the same ensemble. At each training time, this dimension is the number of eigenvalues lying above the image/kernel threshold εtol λ1\varepsilon_{\rm tol}\,\lambda_{1} drawn in Fig. 6 .
Figure 9: The modes q(r)q^{(r)} of the training Hamiltonian, eq. 27 , as functions of xx and shown at three training epochs: τ=0\tau=0 (top block), τ=1000\tau=1000 (middle block), and τ=10000\tau=10000 (bottom block). The eight panels of each block show, for each fitted flavour, the ranks r=1,2,3,5,10r=1,2,3,5,10 coloured as indicated in the legend. The modes are those of a single replica. The vertical scale differs between flavours but is the same across the three epochs. The overall sign of each mode is chosen so that its coefficient in the expansion of f(in)f^{(\rm in)} on the modes is positive. The dashed black line is the central replica of the underlying PDF f(in)f^{(\rm in)} used to generate the data, rescaled for each flavour by the factor indicated, common to the three epochs.
Figure 10: Characteristic resolution times τ(i)=1/(2ηh(i))\tau^{(i)}=1/(2\eta h^{(i)}) , in epochs, of every mode ii in Im(Θ)\mathrm{Im}(\Theta) at the frozen epoch 1500015000 (ordered by decreasing h(i)h^{(i)} ). The threshold separating the image from the kernel is obtained with εtol=10−12\varepsilon_{\rm tol}=10^{-12} , as in Fig. 8 , giving d⊥≃106d^{\perp}\simeq 106 median modes at this epoch. The red dashed line marks the training horizon ( τmax=20000\tau_{\rm max}=20000 epochs). The dotted lines are the three evolution times τ\tau of Fig. 11 . Points are medians over the ensemble of replicas.
Figure 11: The analytical solution (blue) of the eight fitted flavours (rows) computed with the NTK frozen at τNTK=15000\tau_{\rm NTK}=15000 and for five evolution times (columns). The initial condition f0f_{0} of the analytical solution is an independent ensemble of PDFs at initialisation for all panels. In the first three columns the analytical solution is compared against the trained network (orange) evaluated at the same training time, while in the last two columns the trained network is kept fixed at the last available epoch, τmax=20000\tau_{\rm max}=20000 . Curves show the mean, one-sigma band and 68%68\% c.l. over the ensemble of replicas. The underlying law f(in)f^{(\rm in)} is also drawn as a dashed black line.
Figure 12: Analytical (blue) versus empirically trained (orange) evolution of the singlet Σ\Sigma , for three choices of the epoch at which the NTK is frozen ( τNTK=0,5000,10000\tau_{\rm NTK}=0,5000,10000 , rows) and the same three evolution times τ\tau as in Fig. 11 (columns). Statistics and the underlying law (dashed black) are as in Fig. 11 .
Figure 13: Decomposition of the analytical solution of eq. 30 into the contribution from the initial condition, U(t)f0U(t)f_{0} (green), and the contribution from the data, V(t)YV(t)Y (purple, hatched), together with their sum (blue), for the eight fitted flavours (rows) and the five evolution times τ\tau of Fig. 11 (columns). The NTK is frozen at epoch 1500015000 . The initial condition and the statistics over the ensemble of replicas are as in Fig. 11 . The dashed black line is the underlying law f(in)f^{(\rm in)} .
Figure 14: Fractions of the variance of the analytical solution carried by the initial condition (green), by the data (purple), and by their correlation (dotted grey). Each curve is the ratio between the diagonal entries of the corresponding terms in eq. 67 and the total covariance, so that all fractions sum to one at each point in xx . The layout and settings are the same as in Fig. 11 . The two horizontal grey lines mark zero and one of the y-scale. The point at x=1x=1 , where PDFs vanish by construction, is omitted.
Figure 15: One-sigma width σ[xf(x)]\sigma[xf(x)] of the PDFs at initialisation as a function of xx at the initial scale Q0Q_{0} , computed as the standard deviation over Nrep=120N_{\rm rep}=120 replicas of the full model, preprocessing factors included. Each panel shows one of the eight fitted flavours, and each curve one of the six values of the initialisation exponent γ\gamma of eq. 68 . Both axes are logarithmic.
Figure 16: Ratio of the widths of Fig. 15 for consecutive values γ<γ′\gamma<\gamma^{\prime} of the initialisation exponent, one panel per fitted flavour. The horizontal dotted line is the value 1/r=65.211/r=65.21 predicted in the linear regime by eq. 69 . The ratio is close to this value for γ≳3\gamma\gtrsim 3 , whereas for smaller γ\gamma it falls below it, most strongly at small xx , where the log⁡x\log x input is largest and the hyperbolic tangents saturate.
Figure 17: Training (solid) and validation (dashed) losses as functions of the epoch, for a single representative replica. Each curve represents one of the six values of γ\gamma considered in this section. Both axes are logarithmic. For γ≳3\gamma\gtrsim 3 the trajectory develops a plateau, on which the loss stalls close to its initial value, followed by a rapid drop. Increasing γ\gamma delays that drop and slightly lifts the level reached afterwards.
Figure 18: Training trajectories of the leading 5050 eigenvalues λ(r)\lambda^{(r)} of the NTK, with preprocessing factors excluded. Each panel corresponds to a different initialisation exponent γ\gamma of eq. 68 . The colour encodes the eigenvalue rank rr , as in Fig. 6 . The curves are medians over the Nrep=120N_{\rm rep}=120 replicas of each fit, and bands are omitted for clarity; the red dashed line marks the median escape epoch τesc\tau_{\rm esc} of each fit. The abscissa is linear below 5050 epochs and logarithmic above, so that the initialisation at τ=0\tau=0 fits on the axis. The ordinate is cut at 10−1010^{-10} , the smallest eigenvalue that still lies inside Im(Θ)\mathrm{Im}(\Theta) at the tolerance of Fig. 8 for any of these fits.
Figure 19: Displacements of the parameters in Euclidean space as a function of training time and for the different initialisation widths. (Left) The distance ∥θτ−θ0∥\|\theta_{\tau}-\theta_{0}\| separating the parameters from their position at initialisation. (Right) The same distance normalised to the size of the initialisation, ∥θ0∥\|\theta_{0}\| . Only the network parameters are shown, the preprocessing factors being frozen after initialisation. Solid lines are means over the Nrep=120N_{\rm rep}=120 replicas of each fit and the shaded areas are one-sigma bands. The dotted vertical lines mark the median escape epoch τesc\tau_{\rm esc} of each fit. The abscissa is linear below 5050 epochs and logarithmic above, so that the initialisation at τ=0\tau=0 sits on the axis. The first stored epoch is omitted, the displacement vanishing there by construction.
Figure 20: Training loss of the analytical solution of eq. 30 with the NTK frozen at initialisation (dashed) against the loss of the empirically trained network (solid), one panel per initialisation exponent γ\gamma . The ordinate is logarithmic while the abscissa is linear below epoch 5050 and logarithmic above. The analytical loss is evaluated by feeding the evolved solution back through the same FK tables and inverse covariance that define the trained loss, so that the two curves are the same observable. Both are medians over the Nrep=120N_{\rm rep}=120 replicas of each fit. The dotted vertical line marks the median escape epoch τesc\tau_{\rm esc} of the ensemble. The trained curves are those of Fig. 17 , up to the replacement of the single representative replica by the ensemble median.
Figure 21: Analytical (blue) versus empirically trained (orange) solution of the singlet Σ\Sigma , with the NTK frozen at initialisation, τNTK=0\tau_{\rm NTK}=0 . The rows show the analytical solution obtained with γ=1\gamma=1 (top), γ=3\gamma=3 (middle), γ=5\gamma=5 (bottom). The columns represent three different evolution epochs τ\tau common to the three rows. Analytical and trained solutions are compared at the same epoch τ\tau . The solid line represents the mean over the ensemble, the shaded area the one-sigma band and the dashed lines the 68%68\% c.l. over the Nrep=120N_{\rm rep}=120 replicas of each fit. The dashed black line is the underlying law f(in)f^{(\rm in)} used to generate the data.
Figure 22: Same as Fig. 20 , but with the NTK frozen at the last stored epoch, τNTK=τend=2×104\tau_{\rm NTK}=\tau_{\rm end}=2\times 10^{4} .
Figure 23: Same as Fig. 21 , but with the NTK frozen at the last stored epoch, τNTK=τend=2×104\tau_{\rm NTK}=\tau_{\rm end}=2\times 10^{4} .
Figure 24: (Left) Minimum over the first 2×1042\times 10^{4} epochs of the validation loss, minus that of γ=1\gamma=1 for the same replica: mean over 120120 replicas with its bootstrap error (thick bar), and 68%68\% interval of the replicas (thin line). The ordinate is linear within ±0.05\pm 0.05 and logarithmic outside. (Middle) Number of epochs for the training loss to fall below 2.32.3 , divided by that of γ=1\gamma=1 for the same replica: median with its bootstrap 68%68\% interval (thick bar), and 68%68\% interval of the replicas (thin line); the arrow marks γ=5\gamma=5 , where 31%31\% of the replicas never reach 2.32.3 within 2×1042\times 10^{4} epochs. (Right) η hmax⁡\eta\,h_{\max} , with hmax⁡h_{\max} the largest eigenvalue of H⊥H^{\perp} , at initialisation (open) and at the end of the training (filled), median and 68%68\% interval over five replicas. The dotted line represents the stability condition of GD, eq. 73 . We use black circles for the fixed learning rate, η=10−6\eta=10^{-6} , and orange squares for the rescaled learning rates so that η hmax⁡\eta\,h_{\max} at initialisation matches that of γ=1\gamma=1 ( η=10−7\eta=10^{-7} for γ=0.001,0.1\gamma=0.001,0.1 and 2×10−72\times 10^{-7} for γ=0.5\gamma=0.5 ), drawn slightly to the right.
Figure 25: Comparison between the trained solution with the contribution of the data removed, f−V(T)Yf-V(T)Y (black), and the contribution of the initial condition to the analytical solution, U(T)f0U(T)f_{0} (green), for the eight fitted flavours (rows) and for γ=0.5\gamma=0.5 , 0.750.75 , 11 , 1.251.25 and 1.51.5 (columns). For each fit the NTK is frozen at the last epoch, τNTK=τend=2×104\tau_{\rm NTK}=\tau_{\rm end}=2\times 10^{4} , and both quantities are evaluated at the end of the training, with TT corresponding to τ=τend\tau=\tau_{\rm end} . The two coincide only if the analytical solution reproduces the trained network. The solid lines represent the mean over the ensemble, the shaded areas the one-sigma band and the dashed lines the 68%68\% c.l. over the Nrep=120N_{\rm rep}=120 replicas of each fit.
Figure 26: Eigenvalues of the NTK of the neural network (preprocessing factors excluded) at initialisation, against their rank. Coloured curves are the baseline architecture [25,20,8][25,20,8] at γ=1,2,3\gamma=1,2,3 . The black curve is the wide architecture [500,500,8][500,500,8] at γ=1\gamma=1 . Lines are medians over the replicas of each fit ( Nrep=120N_{\rm rep}=120 for the baseline architecture, 55 for the wide one), bands are 68%68\% intervals. Each curve stops at the dimension of the image of the NTK, with the tolerance of Fig. 8 .
Figure 27: Training loss at γ=1\gamma=1 for the baseline architecture [25,20,8][25,20,8] (learning rate η=10−6\eta=10^{-6} ) and the wide architecture [500,500,8][500,500,8] ( η=10−7\eta=10^{-7} ), as a function of the epoch (left) and of the gradient-flow time t=η τt=\eta\,\tau (right). Lines are medians and bands 68%68\% intervals over the first five replicas of each fit. Both axes are logarithmic.
Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 28: Saturation of the first (left) and second (right) hidden layers as a function of xx as defined in eq. 82 . The six values of γ\gamma considered in this analysis are represented by the different coloured curves. The curves are means over the Nrep=120N_{\rm rep}=120 replicas of each fit. For γ≳3\gamma\gtrsim 3 the saturation is negligible across the entire xx range, whereas for smaller values of γ\gamma the saturation becomes significant, particularly at small xx where the log⁡x\log x input is largest.
Figure 29: The spectrum of Θ\Theta against the eigenvalue rank rr at initialisation, for the six initialisation exponents γ\gamma and computed from the neural network alone (no preprocessing). (Left) The eigenvalues normalised to the leading one, so that the six fits share an ordinate. (Right) The gap between consecutive eigenvalues, λr/λr+1\lambda_{r}/\lambda_{r+1} , taken replica by replica before the median. Curves are medians over the Nrep=120N_{\rm rep}=120 replicas and bands are 68%68\% intervals; there is one marker per eigenvalue. The dotted vertical lines mark the three tier boundaries NoutN_{\rm out} , 2Nout2N_{\rm out} and 3Nout3N_{\rm out} predicted by eq. 71 . Note that the curve for each γ\gamma stops at its own dim⁡Im(Θ)\dim\mathrm{Im}(\Theta) of eq. 149 , that is, the last point before Ker(Θ)\mathrm{Ker}(\Theta) .
Figure 30: (Left) The dimension of the image of the NTK, dim⁡Im(Θ)\dim\mathrm{Im}(\Theta) , as a function of the training epoch, for the six values of γ\gamma considered in this analysis. The tolerance is set as in Fig. 8 . Preprocessing factors are excluded from the NTK. The curve with γ=1\gamma=1 is the same as Fig. 8 without preprocessing factors. (Right) The saturation sα(ℓ)s^{(\ell)}_{\alpha} of eq. 82 , averaged over the input grid in xx , as a function of the training epoch. Here shown for the first (solid) and second (dashed) hidden layer, whose values at τ=0\tau=0 are those tabulated in eq. 83 . In both panels, solid lines are medians over the Nrep=120N_{\rm rep}=120 replicas of each fit and the shaded bands are the 68%68\% intervals; the vertical dotted lines mark the median escape epoch τesc\tau_{\rm esc} of the corresponding fit. The abscissa is linear below 5050 epochs and logarithmic above, so that the initialisation at τ=0\tau=0 fits on the axis.
Figure 31: The singlet (upper block) and gluon (lower block) components of six modes q(r)q^{(r)} of the training Hamiltonian, eq. 27 , as functions of xx , for the six values of the initialisation exponent γ\gamma . Within each block the three rows are the epoch at which the mode is evaluated: initialisation, the escape epoch τesc\tau_{\rm esc} of each fit, and the last stored epoch τend=2×104\tau_{\rm end}=2\times 10^{4} , well beyond the drop of the loss for every γ\gamma . The columns are the mode rank, taken as the first and the last of each tier of NoutN_{\rm out} . The modes are computed for the bare network, with the preprocessing factors removed, on a single replica common to the six fits. Each curve is normalised by its own signed value at the point of largest ∣q(r)∣|q^{(r)}| , independently for each flavour and each rank, so that the largest excursion of every curve is +1+1 . Note that in logarithmic scale a straight line is a pure log⁡x\log x dependence.
Figure 32: Residual ρ(r)\rho^{(r)} of eq. 150 as a function of the rank rr , at initialisation (left) and at the escape epoch τesc\tau_{\rm esc} (right). Curves are medians over the Nrep=120N_{\rm rep}=120 replicas of each fit and bands are 68%68\% intervals. The curves for γ=4\gamma=4 and 55 terminate at the median dim⁡Im(Θ)\dim\mathrm{Im}(\Theta) of the corresponding fit ( cfr. Fig. 30 ). The dotted vertical lines mark the boundaries of the tiers as in Fig. 29 . The wide band on the γ=5\gamma=5 curve of the right panel is due to the 2222 replicas out of 120120 that never escape within the training and therefore remain close to Vlin\mathcal{V}_{\rm lin} .
Figure 33: Subspace drift dkd_{k} of eq. 151 , measuring how far the kk leading eigenvectors of Θτ\Theta_{\tau} have rotated away from the subspace they spanned at initialisation, as a function of the training epoch. One panel per subspace size kk , sharing both axes. The first five panels use a value of kk common to all six fits, chosen at the tier boundaries 88 , 1616 and 24=3×824=3\times 8 together with k=1k=1 and k=2k=2 within the first tier. The last panel sets kk to the dimension of the image at initialisation, dim⁡Im(Θ0)\dim\mathrm{Im}(\Theta_{0}) . Curves are medians over the Nrep=120N_{\rm rep}=120 replicas of each fit and bands are 68%68\% intervals; the dotted vertical lines mark the median escape epoch τesc\tau_{\rm esc} . The NTK is that of Fig. 18 , with the neural network alone. The initialisation is omitted, dkd_{k} vanishing there by construction.
Figure 34: Characteristic resolution times τ(i)=1/(2ηh(i))\tau^{(i)}=1/(2\eta h^{(i)}) of eq. 66 , in epochs, of every mode in Im(Θ)\mathrm{Im}(\Theta) with the NTK frozen at initialisation, against the mode rank ii and for the six initialisation exponents γ\gamma . The kernel is that of the full model, as in Fig. 10 , so that each curve terminates at the corresponding entry of eq. 70 . Points are medians over the Nrep=120N_{\rm rep}=120 replicas of each fit. The red dashed line marks the training horizon τend=2×104\tau_{\rm end}=2\times 10^{4} epochs, the last epoch of the numerical training: a mode whose resolution time lies above it has not had the time to be learned. The dotted vertical line marks the Nout=8N_{\rm out}=8 modes of the constant sector. The star on each curve is the slowest mode that the corresponding fit could have resolved by its own escape epoch τesc\tau_{\rm esc} .
Figure 35: The discrepancy DD of eq. 153 between the analytical and the trained solution at the end of the training, as a function of the epoch τNTK\tau_{\rm NTK} at which the NTK is frozen, for the six values of the initialisation exponent γ\gamma . (Left) Against τNTK\tau_{\rm NTK} in epochs, with the median escape epoch τesc\tau_{\rm esc} of each fit marked by a dotted vertical line of the same colour. The abscissa is logarithmic, so that the point at τNTK=0\tau_{\rm NTK}=0 is shown only in the right panel. (Right) The same curves against τNTK\tau_{\rm NTK} in units of the escape epoch of each fit, on a linear abscissa truncated at 2.32.3 . The dashed grey line marks τNTK=τesc\tau_{\rm NTK}=\tau_{\rm esc} . Markers are the sampled freezing epochs, and lines only interpolate between them. DD is computed from the ensemble means over the Nrep=120N_{\rm rep}=120 replicas of each fit.
More in this topic

Neural Network Training Dynamics

Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.

See all 516 papers

Explore similar work

CardsList
Dec 30, 2025

Quantitative Understanding of PDF Fits and their Uncertainties

Parton Distribution Functions (PDFs) play a central role in describing experimental data at colliders and provide insight into the structure of nucleons. As the LHC enters an era of high-precision measurements, a robust PDF determination with a reliable uncertainty quantification has become mandatory in order to match the experimental precision. The NNPDF collaboration has pioneered the use of Machine Learning (ML) techniques for PDF determinations, using Neural Networks (NNs) to parametrise the unknown PDFs in a flexible and unbiased way. The NNs are then trained on experimental data by means of stochastic gradient descent algorithms. The statistical robustness of the results is validated by extensive closure tests using synthetic data. In this work, we develop a theoretical framework based on the Neural Tangent Kernel (NTK) to analyse the training dynamics of neural networks. This approach allows us to derive, under precise assumptions, an analytical description of the neural network evolution during training, enabling a quantitative understanding of the training process. Having an analytical handle on the training dynamics allows us to clarify the role of the NN architecture and the impact of the experimental data in a transparent way. Similarly, we are able to describe the evolution of the covariance of the NN output during training, providing a quantitative description of how uncertainties are propagated from the data to the fitted function. While our results are not a substitute for PDF fitting, they do provide a powerful diagnostic tool to assess the robustness of current fitting methodologies. Beyond its relevance for particle physics phenomenology, our analysis of PDF determinations provides a testbed to apply theoretical ideas about the learning process developed in the ML community.

Amedeo Chiefa, Luigi Del Debbio, Richard Kenway

The Higgs Centre for Theoretical Physics, School of Physics and Astronomy, The University of Edinburgh, Peter Guthrie Tait Road, Edinburgh EH9 3FD, United Kingdom

Uncertainty QuantificationHigh-Energy PhysicsNeural Network Training Dynamics
AskPDF
May 9, 2026

The Global Empirical NTK: Self-Referential Bias and Dimensionality of Gradient Descent Learning

In training a neural network with gradient descent (GD), each iteration induces a linear operator that governs the first-order updates to a model's internal state variables. We call this operator the Global Empirical Neural Tangent Kernel (GeNTK). In finite-width networks, GeNTK is typically intractable to form, leading prior work to focus on restrictive settings, such as tracking only outputs or infinite-width limits. Here, we bridge this gap by studying the structure of GeNTK across many models. Formulating the model state as the solution to a global implicit constraint, we derive GeNTK as a product of two operators: K, accounting for immediate parameter-to-state interactions, and P, describing internal state-to-state dependencies. For a broad class of "weight-based models", such as RNNs and transformers, we prove a universal Kronecker-core theorem showing that K admits an exact, forward-pass-computable form given by the Gram matrix of weight-site variables. This core gives a measurable state-dependent restriction on Euclidean GD, termed "self-referential bias": current activity filters which adjoint directions can drive a first-order update. For recurrent models (GRUs and RNNs), we examine the spectrum of GeNTK and formalize notions of low-rank, biased learning in terms of reduced spatial or temporal ranks of the full operator. Practically, this explains why some RNN initializations yield better learning, while some plateau, by biasing the GeNTK and thus restricting learning to a small domain of learnable tasks. To demonstrate computational viability, we validate the main theorems in a self-attention transformer and a pretrained Pythia transformer language model with 410 million parameters. Overall, our results expose tractable local rank and mode-selective biases in the finite-width GD geometry and open the door to principled study of the GeNTK, a bridge between theory and practice.

James Hazelden, Laura Driscoll, Eli Shlizerman +1

Applied Mathematics, University of Washington · Allen Institute · Neurobiology & Biophysics, University of Washington +1

Representation LearningNeural Tangent KernelGradient Descent Dynamics
AskPDF
Jul 11, 2026

The Differential Neural Tangent Kernel and Its Positivity

The Neural Tangent Kernel (NTK) is one powerful tool for analyzing the training dynamics of neural networks in the over-parameterized regime. Recently, the theoretical framework has been extended to physics-informed neural networks (PINNs) for solving linear PDEs, one highly popular class of neural PDE solvers. In the analysis, the positivity of the associated NTK plays a fundamental role. However, establishing the positivity of the NTK for PINNs is highly challenging, due to the presence of multiple differential operators. In this work, we propose a new theoretical framework, called Differential Neural Tangent Kernel (DNTK), for analyzing PINNs through the lens of the NTK, and establish the positivity of the infinite width DNTK for both shallow and deep neural networks for a wide class of activation functions, including RePU and smooth but non-polynomial activations, for all linear differential operators. These theoretical results lay the foundation for the analysis of gradient type algorithms for training PINNs.

Bangti Jin, Longjun Wu

Department of Mathematics, The Chinese University of Hong Kong, Shatin, N.T., Hong Kong, P.R. China

Neural PDE SolversPhysics-Informed MLNeural Network Training DynamicsNeural Tangent Kernel
AskPDF