In 2020 Domingos introduced an interpolation formula valid for "every model trained by gradient descent". He concluded that such models behave approximately as kernel machines. In this work, we extend the Domingos formula to stochastic training. We introduce a stochastic gradient kernel that extends the deterministic version via a continuous-time diffusion approximation. We prove stochastic Domingos theorems and show that the expected network output admits a kernel-machine representation with optimizer-specific weighting. It reveals that training samples contribute through loss-dependent weights and gradient alignment along the training trajectory. We then link the generalization error to the null space of the integral operator induced by the stochastic gradient kernel. The same path-kernel viewpoint provides a unified interpretation of diffusion models and GANs: diffusion induces stage-wise, noise-localized corrections, whereas GANs induce distribution-guided corrections shaped by discriminator geometry. We visualize the evolution of implicit kernels during optimization and quantify out-of-distribution behaviors through a series of numerical experiments. Our results support a feature-space memory view of learning: training stores data-dependent information in an evolving tangent feature geometry, and predictions at test time arise from kernel-weighted retrieval and aggregation of these stored features, with generalization governed by alignment between test points and the learned feature memory.
Figures & tables
Figure 1: Evolving neighborhoods around anchor points (marked as ”x”). (a) Binary dataset with class y=1 on the circle x12+x22=1 and class y=−1 inside the disk x12+x22≤0.8 . (b) SGD decision regions after training. In (c–f), black crosses denote fixed anchor inputs (coordinates listed in the legend). For each anchor, we highlight its 100 nearest training points under the specified notion of similarity. (c) Neighbors in Euclidean input space, which are largely class mixed. (d–f) Neighbors under the normalized gradient kernel Kt at initialization, the 200th epoch, and the last epoch. Training progressively sharpens these kernel neighborhoods toward label-homogeneous sets.
Figure 2: Prediction and gradient kernel evolution. (a) Learned prediction versus ground truth for the sine function sinx . (c) Learned prediction versus ground truth for the square-wave target. (b,d) Normalized gradient kernel matrices at the final iteration (1000 steps) for sinx and the square wave, respectively. (e–f) MNIST and (g–h) CIFAR-10: normalized gradient kernel matrices at initialization (e,g) and after convergence (f,h). For MNIST/CIFAR-10, we use 10 images per class (100 samples total), ordered by class.
Figure 3: SVM classification in input space versus tangent-feature space. In (a,b), we fit a linear SVM using the learned tangent features ϕ(xn)=∇Θf(xn,ΘT) as inputs (each sample xn is represented by ϕ(xn) , not by its coordinates). In (c,d), we fit an SVM with a Gaussian RBF kernel on the raw inputs. Panels (a,c) use the isotropic circle task x:x12+x22≤0.9 (label −1 ) versus x:x12+x22=1 (label +1 ); panels (b,d) use the anisotropic ellipse variant x:100x12+x22≤0.9 (label −1 ) versus x:100x12+x22=1 (label +1 ).
Figure 4: UMAP visualization of MNIST digits in the tangent feature space {∇Θf(xn,Θk)}n=1N . Left: At initialization ( 0 -th epoch), digits 0 – 4 are not clearly separated, indicating limited discriminative structure. Middle: After training on digits 0 – 4 ( 200 -th epoch), the tangent-space representations of these classes become well clustered and separable. Right: Tangent-feature visualization of all digits: while trained digits ( 0 – 4 ) remain well separated, unseen digits ( 5 – 9 ) fail to form distinct clusters.
Figure 5: Normalized gradient kernel for out-of-domain printed digits. Normalized gradient kernel between MNIST digits k (indices 2k×10 - 2(k+1)×10 ) and printed digits k (blocks at indices (2k+1)×10 - (2k+2)×10 , where k=0,1,⋯,9 ) at initialization (a) and after training ( 200 -th epoch) (b). Samples are ordered by class, with 10 images per class for MNIST digits (0–9) and 10 printed images per digit; the block structure highlights within-digit and cross-digit similarities between the two domains. (c) Confusion matrix for classifying printed digits using the network trained on MNIST.
Figure 6: Normalized gradient kernel for out-of-domain inputs (unseen letters). Normalized gradient kernel between MNIST digits (indices 0-100) and letters M,N,U,V,A (indices 100-150) at initialization (a) and after training ( 300 -th epoch) (b). Each subfigure displays 10 samples per digit (0-9) and 10 letters. The dashed lines indicate the boundary between digit and letter blocks.
Figure 7: Two training settings for the 1-D regression y=2x+1 . In the first column , the green band represents the training regime, the red line is the reference function, and the blue line is the network’s prediction. The second column shows the evolution of bias parameters during training epochs under different initializations. The third column illustrates the model rank for training data (blue) \mboxrank(∇Θf(Xtrain,Θ)) and training-testing mixture data \mboxrank(∇Θf(Xmixture,Θ)) . The fourth column shows the test loss. The bottom row has one bias outside the training domain, so the corresponding neuron is never activated during training. The rank gap \mboxrank(∇Θf(Xmixture,Θ))>\mboxrank(∇Θf(Xtrain,Θ)) emerges and the test loss remains higher, constituting evidence of extrapolation failure.
Figure 8: The visualization for the first layer ReLU activation regimes at convergence. The left panel corresponds to Figure 7 and the right panel to Figure 7 . Colored curves show the responses of neurons with biases bi,i=1,⋯,6 . Inputs fall in dark-gray vanishing region have zero output and zero tangent features. Left : all units are trained within [0,1.98] , no previously unseen tangent directions appear at test time. Right : the parameter b6=2.0 never activated during training but turns on at test time, thus ∇Θf(x)∈/span{∇Θf(xn,Θ)}n=1N producing extrapolation failure.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Figure (a)(b)(c)(d) are the classification results for SGD, SGDM, Adam and RMSprop. The second row shows the nearest 100 points under the tangent space generated by SGD, SGDM, Adam and RMSprop (Figure (e)-(h)).
Figure 10: Evolution of the normalized gradient kernel Kt(x,y)=Kt(x,y)/Kt(x,x)Kt(y,y) over training in the binary classification task in Figure 1 . We fix x=(0,0) with label −1 and track y={(0,0),(0.5,0),(1.0,0),(1.2,0)} , where (0.5,0) has label −1 while y=(1.0,0) and y=(1.2,0) have label +1 . Same-label pairs maintain high NTK similarity (near 1 ), while opposite-label pairs rapidly decay toward 0 and may become negative, indicating persistent gradient anti-alignment across classes. The transient fluctuations and steady-state behaviors differ across (a) SGD, (b) SGDM, (c) RMSprop, and (d) Adam.