On Interpolation Formulas Describing Neural Network Generalization

Jin Guo; Roy Y. He; Jean-Michel Morel

cs.LGMar 14, 2026

On Interpolation Formulas Describing Neural Network Generalization

Authors: Jin Guo, Roy Y. He, Jean-Michel Morel

Organizations: Department of Mathematics, City University of Hong Kong, Hong Kong · Division of Industrial Data Science, Lingnan University, Hong Kong

Abstract

In 2020 Domingos introduced an interpolation formula valid for "every model trained by gradient descent". He concluded that such models behave approximately as kernel machines. In this work, we extend the Domingos formula to stochastic training. We introduce a stochastic gradient kernel that extends the deterministic version via a continuous-time diffusion approximation. We prove stochastic Domingos theorems and show that the expected network output admits a kernel-machine representation with optimizer-specific weighting. It reveals that training samples contribute through loss-dependent weights and gradient alignment along the training trajectory. We then link the generalization error to the null space of the integral operator induced by the stochastic gradient kernel. The same path-kernel viewpoint provides a unified interpretation of diffusion models and GANs: diffusion induces stage-wise, noise-localized corrections, whereas GANs induce distribution-guided corrections shaped by discriminator geometry. We visualize the evolution of implicit kernels during optimization and quantify out-of-distribution behaviors through a series of numerical experiments. Our results support a feature-space memory view of learning: training stores data-dependent information in an evolving tangent feature geometry, and predictions at test time arise from kernel-weighted retrieval and aggregation of these stored features, with generalization governed by alignment between test points and the learned feature memory.

Figures & tables

Panel of Figure 1 — Figure 1: Evolving neighborhoods around anchor points (marked as ”x”). (a) Binary dataset with class $y=1$ on the circle $x_{1}^{2}+x_{2}^{2}=1$ and class $y=-1$ inside the disk $x_{1}^{2}+x_{2}^{2}\leq 0.8$ . (b) SGD decision regions after training. In (c–f), black crosses denote fixed anchor inputs (coordinates listed in the legend). For each anchor, we highlight its $100$ nearest training points under the specified notion of similarity. (c) Neighbors in Euclidean input space, which are largely class mixed. (d–f) Neighbors under the normalized gradient kernel $\widehat{K}_{t}$ at initialization, the 200th epoch, and the last epoch. Training progressively sharpens these kernel neighborhoods toward label-homogeneous sets.

Panel of Figure 2 — Figure 2: Prediction and gradient kernel evolution. (a) Learned prediction versus ground truth for the sine function $\sin x$ . (c) Learned prediction versus ground truth for the square-wave target. (b,d) Normalized gradient kernel matrices at the final iteration (1000 steps) for $\sin x$ and the square wave, respectively. (e–f) MNIST and (g–h) CIFAR-10: normalized gradient kernel matrices at initialization (e,g) and after convergence (f,h). For MNIST/CIFAR-10, we use 10 images per class (100 samples total), ordered by class.

Panel of Figure 3 — Figure 3: SVM classification in input space versus tangent-feature space. In (a,b), we fit a linear SVM using the learned tangent features $\phi(x_{n})=\nabla_{\mathbf{\Theta}}f(x_{n},\mathbf{\Theta}_{T})$ as inputs (each sample $x_{n}$ is represented by $\phi(x_{n})$ , not by its coordinates). In (c,d), we fit an SVM with a Gaussian RBF kernel on the raw inputs. Panels (a,c) use the isotropic circle task $x:x_{1}^{2}+x_{2}^{2}\leq 0.9$ (label $-1$ ) versus $x:x_{1}^{2}+x_{2}^{2}=1$ (label $+1$ ); panels (b,d) use the anisotropic ellipse variant $x:100x_{1}^{2}+x_{2}^{2}\leq 0.9$ (label $-1$ ) versus $x:100x_{1}^{2}+x_{2}^{2}=1$ (label $+1$ ).

Figure 4: UMAP visualization of MNIST digits in the tangent feature space $\{\nabla_{\mathbf{\Theta}}f(x_{n},\mathbf{\Theta}_{k})\}_{n=1}^{N}$ . Left: At initialization ( $0$ -th epoch), digits $0$ – $4$ are not clearly separated, indicating limited discriminative structure. Middle: After training on digits $0$ – $4$ ( $200$ -th epoch), the tangent-space representations of these classes become well clustered and separable. Right: Tangent-feature visualization of all digits: while trained digits ( $0$ – $4$ ) remain well separated, unseen digits ( $5$ – $9$ ) fail to form distinct clusters.

Panel of Figure 5 — Figure 5: Normalized gradient kernel for out-of-domain printed digits. Normalized gradient kernel between MNIST digits $k$ (indices $2k\times 10$ - $2(k+1)\times 10$ ) and printed digits $k$ (blocks at indices $(2k+1)\times 10$ - $(2k+2)\times 10$ , where $k=0,1,\cdots,9$ ) at initialization (a) and after training ( $200$ -th epoch) (b). Samples are ordered by class, with 10 images per class for MNIST digits (0–9) and 10 printed images per digit; the block structure highlights within-digit and cross-digit similarities between the two domains. (c) Confusion matrix for classifying printed digits using the network trained on MNIST.

Panel of Figure 6 — Figure 6: Normalized gradient kernel for out-of-domain inputs (unseen letters). Normalized gradient kernel between MNIST digits (indices 0-100) and letters $M,N,U,V,A$ (indices 100-150) at initialization (a) and after training ( $300$ -th epoch) (b). Each subfigure displays 10 samples per digit (0-9) and 10 letters. The dashed lines indicate the boundary between digit and letter blocks.

Panel of Figure 7 — Figure 7: Two training settings for the 1-D regression $y=2x+1$ . In the first column , the green band represents the training regime, the red line is the reference function, and the blue line is the network’s prediction. The second column shows the evolution of bias parameters during training epochs under different initializations. The third column illustrates the model rank for training data (blue) $\mbox{rank}(\nabla_{\mathbf{\Theta}}f(\bm{X}_{train},\mathbf{\Theta}))$ and training-testing mixture data $\mbox{rank}(\nabla_{\mathbf{\Theta}}f(\bm{X}_{mixture},\mathbf{\Theta}))$ . The fourth column shows the test loss. The bottom row has one bias outside the training domain, so the corresponding neuron is never activated during training. The rank gap $\mbox{rank}(\nabla_{\mathbf{\Theta}}f(\bm{X}_{mixture},\mathbf{\Theta}))>\mbox{rank}(\nabla_{\mathbf{\Theta}}f(\bm{X}_{train},\mathbf{\Theta}))$ emerges and the test loss remains higher, constituting evidence of extrapolation failure.

Panel of Figure 8 — Figure 8: The visualization for the first layer ReLU activation regimes at convergence. The left panel corresponds to Figure 7 and the right panel to Figure 7 . Colored curves show the responses of neurons with biases $b_{i},i=1,\cdots,6$ . Inputs fall in dark-gray vanishing region have zero output and zero tangent features. Left : all units are trained within $[0,1.98]$ , no previously unseen tangent directions appear at test time. Right : the parameter $b_{6}=2.0$ never activated during training but turns on at test time, thus $\nabla_{\mathbf{\Theta}}f(x)\notin\mathrm{span}\{\nabla_{\mathbf{\Theta}}f(x_{n},\mathbf{\Theta})\}_{n=1}^{N}$ producing extrapolation failure.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure 9: Figure (a)(b)(c)(d) are the classification results for SGD, SGDM, Adam and RMSprop. The second row shows the nearest 100 points under the tangent space generated by SGD, SGDM, Adam and RMSprop (Figure (e)-(h)).

Panel of Figure 10 — Figure 10: Evolution of the normalized gradient kernel $\widehat{K}_{t}(x,y)=K_{t}(x,y)/\sqrt{K_{t}(x,x)K_{t}(y,y)}$ over training in the binary classification task in Figure 1 . We fix $x=(0,0)$ with label $-1$ and track $y=\{(0,0),(0.5,0),(1.0,0),(1.2,0)\}$ , where $(0.5,0)$ has label $-1$ while $y=(1.0,0)$ and $y=(1.2,0)$ have label $+1$ . Same-label pairs maintain high NTK similarity (near $1$ ), while opposite-label pairs rapidly decay toward $0$ and may become negative, indicating persistent gradient anti-alignment across classes. The transient fluctuations and steady-state behaviors differ across (a) SGD, (b) SGDM, (c) RMSprop, and (d) Adam.

Neural Network Generalization

Neural network generalization studies why trained models perform well on unseen data and how architecture, data, and training shape that ability.

See all 303 papers

Jun 5, 2026

Second-Order Path Kernel Interpolation Formulas in Machine Learning

Understanding how training data shape neural network predictions is a central problem in modern learning theory. In 2020, Pedro Domingos proposed an interpolation formula valid for every model learned by deterministic gradient descent. It expresses the model's prediction as an integral, along the optimization path, of a data-dependent kernel that aligns the model's gradients at the test and training data. Such a first-order characterization remains valid for models trained with batch-based stochastic optimization. In this paper, we develop second-order forms of these interpolation formulas. We show that the leading path-kernel interpolation is supplemented by a curvature-weighted interpolation term. For stochastic gradient descent, an additional sampling-induced component appears, coupling the curvature of the prediction with the covariance of mini-batch gradient noise. We also extend the representation to stochastic gradient descent with momentum, where the interpolation structure is preserved but with the weights modified by a memory-related factor. Moreover, we establish a concentration estimate for the terminal prediction, identifying the fluctuation scale around the expected second-order representation. Together, these results provide a refinement of the path-kernel interpretation of neural network prediction.

Jin Guo, Roy Y. He, Jean-Michel Morel

Department of Mathematics, City University of Hong Kong, Hong Kong · Division of Industrial Data Science, Lingnan University, Hong Kong

Neural Network Training Dynamics

Ask PDF

Feb 23, 2026

Generative Modeling via Kernelized Stochastic Interpolants

We develop a kernel method for generative modeling within the stochastic interpolant framework, replacing neural network training with linear systems. The drift of the generative SDE is

\hat b_t(x) = \nablaφ(x)^\topη_t

, where

η_t \in \mathbb{R}^P

solves a

P\times P

system computable from data, with

P

independent of the data dimension

d

. Since estimates are inexact, the diffusion coefficient

D_t

affects sample quality; the optimal

D_t^*

from Girsanov diverges at

t=0

, but this poses no difficulty and we develop an integrator that handles it seamlessly. The framework accommodates diverse feature maps: scattering transforms, pretrained generative models, etc, enabling generation and model combination without neural network training. We demonstrate the approach on financial time series, turbulence, and image generation.

Florentin Coeurdoux, Etienne Lempereur, Nathanaël Cuvelle-Magar +2

Capital Fund Management, Paris, France · Département d’informatique, ENS, Université PSL, Paris, France · Collège de France, Paris, France +2

Stochastic Differential Equations Stochastic Interpolants Generative Modeling Kernel Methods

Ask PDF

May 2, 2026

A Theory of Generalization in Deep Learning

We present a non-asymptotic theory of generalization in deep learning where the empirical neural tangent kernel partitions the output space. In directions corresponding to signal, error dissipates rapidly; in the vast orthogonal dimensions corresponding to noise, the kernel's near-zero eigenvalues trap residual error in a test-invisible reservoir. Within the signal channel, minibatch SGD ensures that coherent population signal accumulates via fast linear drift, while idiosyncratic memorization is suppressed into a slow, diffusive random walk. We prove generalization survives even when the kernel evolves

\mathcal{O}(1)

in operator norm, the full feature-learning regime. This theory naturally explains disparate phenomena in deep learning theory, such as benign overfitting, double descent, implicit bias, and grokking. Lastly, we derive an exact population-risk objective from a single training run with no validation data, for any architecture, loss, or optimizer, and prove that it measures precisely the noise in the signal channel. This objective reduces in practice to an SNR preconditioner on top of Adam, adding one state vector at no extra cost; it accelerates grokking by

5 \times

, suppresses memorization in PINNs and implicit neural representations, and improves DPO fine-tuning under noisy preferences while staying

3 \times

closer to the reference policy.

Elon Litman, Gabe Guo

Stanford University

Neural Network Generalization Benign Overfitting Neural Network Training Dynamics Neural Tangent Kernel

Ask PDF

Abstract

Neural Network Generalization

Explore similar work

Second-Order Path Kernel Interpolation Formulas in Machine Learning

Generative Modeling via Kernelized Stochastic Interpolants

A Theory of Generalization in Deep Learning