Neural Tangent Kernel

Also known as NTK

Follow this topic Ask about this topic

Momentum

4 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 32

Sep 30, 2026cs.LG

Certified Approximation for Interpretable Representer Landmarks

Representer explanations rank the training landmarks that most influence a self-supervised representation. At scale, this ranking rests on up to four stacked approximations of the empirical neural tangent kernel (eNTK). These are random output heads, a parameter sketch, landmark sampling and a coefficient fit. Existing analyses bound each approximation separately, but none certifies the top-

K

set against their combined error. We introduce CAIRN (Certified Approximation for Interpretable Representer laNdmarks), a framework that carries this error through to the ranking. We derive the exact variance of the sketched multi-head eNTK, which matches measurement within

4\%

where Johnson-Lindenstrauss bounds err by up to

2.5\times

. This yields a high-probability top-

K

certificate for a fixed coefficient fit, alongside exact residual-trace certificates for discarded spectral mass. An exact product-variance identity separates kernel error from fit variability and identifies when a larger kernel budget can still sharpen a ranking. Stochastic Lanczos Quadrature (SLQ) estimates the effective dimension within

0.72\%

and guides the landmark budget without dense eigendecomposition. We show that residual mass does not control class coverage, and residual-greedy selection cuts the worst coverage excess of

k

-means++ from

8.5\times

to

1.55\times

(

4\times

on the sketched eNTK). Cross-view initializers outperform principal-component initialization in five (AUI) to all six (CSI) settings. Against the KREPES Gauss-Newton solver, CAIRN converges

2.5

to

11.3\times

faster, trails by at most

0.31

points and gains up to

3.14

points on MNIST. Together, these results make the reliability of representer explanations measurable and show where approximation budgets are best spent.

Jayanta Mukherjee, Shourya Verma, Mengbo Wang +2

Department of Computer Science Purdue University West Lafayette, IN 47907, USA

Neural Network Interpretability Kernel Methods Neural Tangent Kernel

Sep 28, 2026cs.LG

First Learn, Then Memorize: The Spectral Bias of Diffusion Models

Diffusion models trained on a finite dataset first learn to generate novel, high-quality samples and only much later collapse onto their training set. We identify the mechanism behind this separation of timescales and the object that probes it. The training dynamics of the score function are governed---exactly, and at any width---by the Gram matrix of the Neural Tangent Kernel (NTK) evaluated on the noisy training data, so the timescales of generalization and of memorization must be encoded in its spectrum. We show that they are, and that the structure responsible has no analogue in standard kernel settings. The use of multiple noise realizations per sample (

m

noised copies at a fixed noise level) in the score-matching loss is what restructures the Gram matrix spectrum into two distinct parts. The first, of large eigenvalues, carries the global features of the target distribution and is present already for

m=1

. The second, which the repeated noising creates, consists of the smallest eigenvalues and is supported on eigenvectors aligned with the sample-specific noise directions; it sets a memorization timescale parametrically larger in the training set size

n

. We establish this picture on two fronts. Analytically, we solve the spectrum in the lazy high-dimensional limit for both linear (

n \asymp d

) and polynomial (

n \asymp d^k

) sample complexities, and prove through a bias--variance decomposition that the first bulk minimizes the approximation error while the second drives the error associated with memorization. Empirically, we show the same two-bulk structure in Convolutional NTKs on CelebA and in finite-width U-Nets trained well beyond the lazy regime, and we make the link causal: truncating the Gram matrix at rank

r

tunes the generalization--memorization transition, and an

L_2

penalty targeting the second bulk suppresses memorization in feature-learning U-Nets.

Raphaël Urfin, Tony Bonnaire, Giulio Biroli +1

Laboratoire de Physique de l’École normale supérieure, ENS, Université PSL, CNRS, Sorbonne Université, Université Paris Cité, F-75005 Paris, France · Université Paris-Saclay, CNRS, Institut d’Astrophysique Spatiale, 91405 Orsay, France · Department of Computing Sciences, Bocconi University, Milano, Italy

Neural Network Generalization Memorization in Generative Models Neural Network Training Dynamics Neural Tangent Kernel Diffusion Models

Sep 22, 2026cs.LG

A Spectral Theory of Grokking: Weight Decay induces Feature Learning

In grokking an early fit to the training data separates from a much later improvement in generalization. During this delay, training can move from a fixed neural tangent kernel (NTK) regime to one in which task-relevant kernel eigendirections continue to evolve. We provide a quantitative theory for how this transition from lazy to rich learning can produce delayed generalization. For homogeneous networks trained with squared loss and

L_2

weight decay, we show that a finite residual remains after memorization, with larger residual fractions in target components associated with smaller NTK eigenvalues. These residuals feed back into the dynamics of the NTK itself, and projecting the resulting dynamics onto task-relevant spectral directions yields a reduced system in which residual-driven kernel growth competes with weight decay. This system predicts that the grokking timescale is controlled by the product of learning rate and weight decay, that feature learning slows logarithmically near a critical decay above which task-aligned NTK structure can no longer support generalization, and that stronger decay can prevent fitting altogether. We test these predictions in modular addition. In a homogeneous MLP, task-aligned Fourier structure continues to emerge in the NTK after training accuracy has saturated, and an 84

\times

90-grid of trained networks across varying learning rate and weight decay recovers the predicted phase geometry and inverse-product scaling of the generalization time with learning rate and weight decay. A one-block Transformer shows similar macroscopic phase structure in a 42

\times

45-grid, as well as the same transition-time scaling despite violating exact homogeneity. Together, these results provide a mechanistic derivation connecting post-fit feature learning to both the onset of generalization and its phase structure in the learning rate and weight decay plane.

Lenz Pracher, Pascal de Jong, Oskar Lieshaus +2

Arnold-Sommerfeld-Center for Theoretical Physics, Ludwig-Maximilians-Universität München, Munich, Germany · Department of Applied Physics, Standford University, Standford, CA, USA · Department of Mathematics, University of Cambridge, Cambridge, UK

Neural Network Generalization Weight Decay Neural Tangent Kernel Grokking

Sep 7, 2026cs.LG

Latent-MoE: Domain-Aware Mixture-of-Experts for PDEs with Multi-Regime Physics

Physics-informed neural networks (PINNs) struggle on PDEs whose governing physics varies across the domain. We trace this to a structural property of standard coordinate networks: their neural tangent kernel (NTK) is translation-variant and lets training points of large coordinate magnitude disproportionately influence predictions elsewhere, producing long-range coupling and gradient conflict during training. We show analytically and empirically that mixture-of-experts (MoE) architectures with centered, compact-support routers yield a uniformly banded NTK whose kernel-regression weights decay exponentially with distance, localizing the learning. Building on this, we propose \emph{Latent-MoE}, which interleaves domain-aware MoE blocks within a shared backbone. Unlike FB-PINNs or X-PINNs, which rigidly partition both the domain and the parameters so that the parameters on different subdomains are updated independently, Latent-MoE is designed to preserve the localization benefit of domain-aware routing while allowing capacity to flow across regions through the shared backbone. On standard homogeneous-physics benchmarks Latent-MoE is competitive with established baselines; on benchmarks with multi-stage time-variable physics, where global models and rigid domain decompositions both fall into spurious solutions, it improves over them by more than an order of magnitude, with markedly reduced gradient conflict during training.

Hanwen Wang, Paris Perdikaris

Graduate Group of Applied Mathematics and Computational Science University of Pennsylvania Philadelphia, PA 19104 · Department of Mechanical Engineering and Applied Mechanics University of Pennsylvania Philadelphia, PA 19104

Neural PDE Solvers Physics-Informed ML Mixture-of-Experts Models Neural Tangent Kernel

Sep 2, 2026cs.LG

Kernel Reboot: Breaking the Boundaries of Neural Tangent Kernels for Neural Fields

Neural fields (NFs) map continuous coordinates to signals such as color or density, but fast high-quality reconstruction from sparse observations remains difficult. Classical Neural Tangent Kernel (NTK) regression gives closed-form fits, yet it is fundamentally linear and cannot accumulate reusable task priors. We develop three algorithms that address these gaps. NTK-KIP learns a distilled support set of coordinates (and optional labels) so that a finite NTK can inpaint large missing regions from little observed data, yielding a compact non-linear representation instead of a raw kernel solve. MetaQuill meta-learns a shared initialization for an INR so that new scenes can be adapted by updating only a small task-specific weight offset, which provides true feature learning and a reusable prior. Finally, MetaQuill-KIP fuses both ideas: it seeds the task with a KIP-style non-linear warm start, then refines only that small offset around the meta-learned initialization. MetaQuill-KIP achieves high-PSNR reconstructions and semantically plausible inpainting under very sparse observations, while requiring only lightweight per-instance adaptation, whereas diffusion-style baselines typically depend on large pretrained generative priors and costly per-image tuning. This shows that NTK-driven neural fields can be made both non-linear and meta-learnable, narrowing the gap between analytic kernels and practical few-shot reconstruction.

Amir Mallak, Alaa Maalouf, Lior Wolf +2

Department of Computer Science, University of Haifa · Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology · School of Computer Science and AI, Tel Aviv University

Implicit Neural Representations Meta-Learning Image Reconstruction Image Inpainting Neural Tangent Kernel

Aug 31, 2026cs.LG

AdaptNTK: Adaptive Uncertainty Quantification and Active Learning for Neural Network Potentials

Machine learning interatomic potentials bridge the gap between quantum chemical precision and classical computational speed, enabling molecular dynamics simulations with first-principles accuracy. Their reliability is often improved through active learning, which iteratively expands the training set by identifying uncertain, out-of-distribution configurations. Existing uncertainty-quantification methods often involve a trade-off between computational cost and reliability, and generally cannot account for redundancy as an acquisition batch is assembled. Here, we introduce AdaptNTK, a single-model framework that measures uncertainty as a regularized Mahalanobis distance in empirical neural tangent kernel (NTK) feature space. With the NTK features fixed during acquisition, the uncertainty depends on the acquired configurations but not their reference labels. This allows the uncertainty to be updated recursively after each selection without retraining, reducing redundancy within an acquisition batch. On held-out rMD17 data, AdaptNTK achieves the highest mean correlations with force errors (Spearman 0.68, Pearson 0.71) and matches a three-member ensemble in error retention. In active learning experiments, AdaptNTK achieves the lowest force errors across rMD17 and Transition-1X, with particularly strong performance on transition-state configurations in Transition-1X. AdaptNTK provides a 2.6-fold speedup per Transition-1X cycle relative to the ensemble, providing efficient single-model uncertainty estimation with sequential updates for data-efficient active learning.

Prajwal Ananth, Shuwen Yue

Center for Applied Mathematics, Cornell University, Ithaca, NY, USA · R. F. Smith School of Chemical and Biomolecular Engineering, Cornell University, Ithaca, NY, USA

Uncertainty Quantification Machine Learning Interatomic Potentials Active Learning Neural Tangent Kernel

Aug 8, 2026cs.LG

Correlation flow governs learning at criticality

The initialization of deep neural networks determines whether information and gradients can propagate across depth, yet a unified theory connecting these properties to learning dynamics remains elusive. Combining mean-field theory and random matrix theory, we establish a direct link between correlation propagation and the Neural Tangent Kernel (NTK) that governs learning in the sequential limit of infinitely wide, infinitely deep networks. Correlation propagation to infinite depth is possible only at a single, critical point in the weight-bias variance plane. At this point, we leverage the algebraic decay of the end-to-end Jacobian with depth to prove that the NTK becomes exactly proportional to the output correlation at infinite depth, tying together information propagation and learning dynamics. We further show that orthogonal initialization suppresses the leading finite-size corrections present under Gaussian initialization, clarifying the respective roles of the two initialization ensembles in this limit. These theoretical predictions are validated quantitatively on finite-width, finite-depth networks. Together, these results demonstrate that orthogonal initialization and criticality are required to control the asymptotic dynamics of deep learning.

Andrea Combette, Nelly Pustelnik, Antoine Venaille

CNRS, ENS de Lyon, Laboratoire de Physique UMR5672, Lyon, France

Neural Network Initialization Neural Network Training Dynamics Neural Tangent Kernel

Aug 4, 2026cs.LG

Tight Worst-Case Bounds for the Smallest Eigenvalue of ReLU NTK Gram Matrices

For

n

unit vectors

x_1,\ldots,x_n \in \mathbb{R}^d

, we study the continuous ReLU derivative Gram matrix

H

, whose entries are obtained by averaging pairwise gated inner products over a standard Gaussian direction. Writing

Δ_\pm := \min_{i \neq j} \min\{ \|x_i-x_j\|_2, \|x_i+x_j\|_2 \}

for their projective separation, we prove the universal dimension-free lower bound

λ_{\min}(H) = Ω( Δ_\pm/\sqrt{\log n} )

. Conversely, we construct worst-case families satisfying the matching upper bound

λ_{\min}(H) = O( Δ_\pm/\sqrt{\log n} )

, showing that this rate is tight up to universal constants.

Zhao Song

ReLU Neural Networks Spectral Methods Neural Tangent Kernel

Jul 11, 2026cs.LG

The Differential Neural Tangent Kernel and Its Positivity

The Neural Tangent Kernel (NTK) is one powerful tool for analyzing the training dynamics of neural networks in the over-parameterized regime. Recently, the theoretical framework has been extended to physics-informed neural networks (PINNs) for solving linear PDEs, one highly popular class of neural PDE solvers. In the analysis, the positivity of the associated NTK plays a fundamental role. However, establishing the positivity of the NTK for PINNs is highly challenging, due to the presence of multiple differential operators. In this work, we propose a new theoretical framework, called Differential Neural Tangent Kernel (DNTK), for analyzing PINNs through the lens of the NTK, and establish the positivity of the infinite width DNTK for both shallow and deep neural networks for a wide class of activation functions, including RePU and smooth but non-polynomial activations, for all linear differential operators. These theoretical results lay the foundation for the analysis of gradient type algorithms for training PINNs.

Bangti Jin, Longjun Wu

Department of Mathematics, The Chinese University of Hong Kong, Shatin, N.T., Hong Kong, P.R. China

Neural PDE Solvers Physics-Informed ML Neural Network Training Dynamics Neural Tangent Kernel

Jul 7, 2026stat.ML

A Function-Space Dichotomy for Compositional Learning: Exponential Sub-Optimality of the Neural Tangent Kernel

A persistent empirical observation is that trained neural networks outperform their neural tangent kernel (NTK) limit on tasks with compositional structure, yet a quantitative account of

\textbf{when}

and

\textbf{by how much}

has been lacking. Working on the unit circle, we give such an account through a dichotomy between two complexity measures of the target: its

\textbf{Fourier complexity}

, which controls NTK kernel regression, and its

\textbf{architectural complexity}

, which controls learning over depth-

L

, width-

w

ReLU networks with the variation norm of the weights bounded by

R

. We first characterize the minimax rate of the architecture class

\mathcal{C}_{L,w,R}

, pinning it down up to a single factor of

L

: between

Ω(Lw^2R^2/n)

and

\tilde{O}(L^2w^2R^2/n)

. We then show the NTK estimator sits

\textbf{exponentially}

above this floor whenever the two complexities decouple: for the depth-

L

iterated sawtooth, NTK regression needs

Ω(4^L)

samples while the minimax floor is polynomial in

L

. Numerical experiments confirm the theoretical claims: on bandlimited smooth targets, the NTK is competitive or better, while on the hypercube sparse-parity model, a standard two-layer network beats the NTK by four to six orders of magnitude in test error. The gap is thus a function-space property, a mismatch between the kernel's smoothness bias and the target's compositional structure, rather than a generic kernel-versus-network phenomenon.

Arkaprabha Ganguli, Emil Constantinescu

Mathematics & Computer Science Division, Argonne National Laboratory, Lemont, IL, USA

Neural Network Generalization Kernel Regression ReLU Neural Networks Neural Network Approximation Theory Neural Tangent Kernel

Jun 1, 2026cs.LG

Scalable Uncertainty Quantification for Extreme Weather Forecasting via Empirical Neural Tangent Kernels

Deep learning weather models now match numerical weather prediction accuracy while running orders of magnitude faster, but produce deterministic forecasts without uncertainty estimates, a critical gap for high-stakes decisions during extreme weather events. This paper proposes Neural Tangent Kernel-based uncertainty quantification (NTK-UQ) using last-layer empirical features. Theoretical analysis predicts that UQ quality is architecture-dependent through two mechanisms. First, a variance collapse mechanism explains when UQ fails: when the eigenvalue truncation rank approaches the effective rank of the feature space, the GP correction term consumes nearly all prior variance, destroying discrimination between tropical cyclones and routine conditions; architectures with concentrated spectra (spectral operators) require aggressive truncation (

k \leq 10

), while attention-based models tolerate full-rank computation. Second, decomposition performance depends on the non-Gaussian, heavy-tailed structure of extreme weather: Independent Component Analysis exploits higher-order statistics (kurtosis, negentropy) to isolate heavy-tailed extreme-event features, achieving higher discrimination than singular value decomposition, which captures only second-order variance. A data-driven selection rule chooses ICA or SVD from the feature eigenspectrum concentration ratio, correctly prescribing the superior decomposition for all four evaluated architectures. Compared to split conformal prediction (the natural post-hoc baseline), NTK-UQ achieves 31--37% sharper prediction intervals at 90% coverage, and uniquely produces \emph{adaptive} intervals that scale with extreme event severity, which conformal prediction cannot achieve by construction. The framework requires no retraining; inference-time uncertainty requires only a single matrix-vector product per sample.

Jose Marie Antonio Miñoza, Rex Gregor Laylo, Sebastian C. Ibañez

Center for AI Research · Department of Education · Makati, Philippines

Uncertainty Quantification Independent Component Analysis Neural Tangent Kernel Regression Prediction Intervals Weather Forecasting

May 29, 2026cs.LG

Spectral Reach: Understanding Neural Scaling as Progress into the Spectral Tail

Neural scaling laws describe predictable power-law relationships between model size, dataset size, compute, and performance. While these laws guide the development of modern foundation models, the mechanisms underpinning them remain poorly understood, in part due to the absence of scalable analysis tools. To close this gap, we introduce "spectral position": a scalable measure of which eigenvalues of the empirical neural tangent kernel (eNTK) currently drive loss reduction. Applying this measure to scaling experiments, we find that spectral position decreases throughout training: learning shifts from dominant eigenmodes into the spectral tail. Larger models reach further into the tail than smaller models, revealing a size-dependent capacity we call "spectral reach". This suggests why larger models achieve lower losses: they sustain learning on weak spectral signals inaccessible to smaller models. We further identify feature learning as a key enabler of spectral reach. It adaptively amplifies gradient magnitudes as learning advances, sustaining progress where frozen representations stall. This points to concrete interventions through architecture and optimizer design.

Konstantin Nikolaou, Jonas Scheunemann, Sven Krippendorf +2

Institute for Computational Physics, University of Stuttgart, Stuttgart, Germany · Department of Applied Mathematics and Theoretical Physics and Department of Physics, University of Cambridge, Cambridge, UK

Representation Learning Neural Network Training Dynamics Neural Tangent Kernel Neural Scaling Laws

May 24, 2026cs.LG

Label-NTK Alignments and A Tighter Convergence Bound in the NTK Regime

The Neural Tangent Kernel (NTK) framework explains optimization in over-parameterized neural networks via approximately linearized dynamics, yielding exponential convergence guarantees. However, existing results are often overly pessimistic and do not match the fast training in practice, as they depend on the smallest NTK eigenvalue, which is typically extremely small in practice. In this work, we develop sharper convergence guarantees by characterizing the interaction between data labels and the NTK eigen-spectrum. We identify two key phenomena, Label-NTK alignment and Residual-NTK alignment, showing that projections of labels and residuals onto NTK eigenvectors scale with the corresponding eigenvalues. We provide empirical evidence and theoretical justification under mild data assumptions. Exploiting these alignment properties, we derive a refined convergence bound that depends on the full spectrum and closely matches practical training dynamics, significantly improving over classical worst-case results. We further obtain improved generalization bounds. Experiments on MLPs and CNNs across multiple datasets validate our theory.

Ruchirinkil Marreddy, Chaoyue Liu

Elmore Family School of Electrical and Computer Engineering Purdue University

Neural Network Generalization Neural Tangent Kernel

May 22, 2026cs.LG

Coupling-Robust Accuracy in Multiphysics Physics Informed Neural Networks via Kronecker-Preconditioned Optimization

Physics-informed neural networks (PINNs) for coupled multiphysics systems suffer systematic accuracy degradation as inter-equation coupling strengthens. We provide a theoretical explanation through neural tangent kernel (NTK) analysis: for linearly coupled systems, the standard NTK's spectral radius grows as

Ω(γ^2)

with coupling strength

γ

, shrinking the stable learning rate, while block-diagonal Gauss--Newton (GN) preconditioning yields a preconditioned NTK

K_P = JH^{+}J^\top

whose spectral radius is bounded by

S

(number of networks), independent of

γ

. Adam's diagonal preconditioning destroys this projector structure -- inflating

λ_{\max}

far above

S

for any coupling type -- and its residual-dynamics kernel grows as

Θ(γ)

, placing its stable learning rate strictly between gradient descent and GN. For one-way coupling the limitation is class-wide: no diagonal preconditioner, fixed or adaptive, halves the driving residual in fewer than

Ω(γ)

iterations (

Ω(γ^2)

if fixed), whereas block-diagonal GN requires

O(1)

. We verify

Ω(γ^2)

growth across linearly coupled benchmarks and confirm

λ_{\max}(K_P) = S

in all three 1D systems, including nonlinearly coupled NP+P. Combining the Kronecker-preconditioned optimizer SOAP with inverse-gradient-norm loss balancing (SOAP+GradNorm) yields coupling-robust accuracy: across 222 experiments spanning three 1D systems and a 2D electroosmotic flow benchmark, SOAP+GradNorm maintains final-epoch

L_2

accuracy across coupling strengths, with

\leq 2.3\times

degradation in nonlinear NP+P while Adam+GradNorm fails (

L_2 > 0.1

). SOAP+GradNorm further scales to a 2D, 6-PDE electroosmotic flow at EDL-resolved conditions down to

\varepsilon = 0.01

-- a regime all prior PINN electrokinetics studies have avoided -- where Adam+GradNorm fails entirely (

L_2 > 0.3

).

Youngjae Park, Jaemin Kim, Junghwa Hong

Dept. of Control and Instrumentation Engineering, Korea University, Sejong, South Korea · BK21 FOUR Smart Mobility Education and Research Team, Korea University, Sejong, South Korea

Gauss-Newton Optimization Neural Network Optimization Physics-Informed ML Neural Tangent Kernel Physics-Based Simulation

May 17, 2026math.OC

Training Infinitely Deep and Wide Transformers

Transformers have become the dominant architecture in modern machine learning, yet the theoretical understanding of their training dynamics remains limited. This paper develops a rigorous mathematical framework for analyzing gradient-based training of transformers in the mean-field regime, where both the depth (number of layers) and width (number of attention heads) tend to infinity. While ResNet training can be understood as controlling a neural ODE, transformer training corresponds to controlling a neural PDE, due to the coupling of multiple token distributions through the attention mechanism. Our mean-field model features two types of measure representations: token distributions evolving through layers and attention parameters at each layer. We establish well-posedness of the forward pass through infinitely deep transformers, characterizing token evolution via flow maps that satisfy ODEs in function spaces. Using adjoint sensitivity analysis, we derive an explicit formula for the conditional Wasserstein gradient of the training risk, involving adjoint variables governed by backward ODEs. We prove the existence and uniqueness of gradient flow curves in the conditional Wasserstein metric space, establishing a rigorous foundation for gradient-based transformer training. A key technical contribution is providing necessary and sufficient conditions for injectivity of the Neural Tangent Kernel (NTK) for attention mechanisms: we show that NTK injectivity is equivalent to linear independence of log-sum-exp functions modulo affine functions, a condition satisfied by diverse token distributions, including discrete distributions, uniform distributions, and Gaussian mixtures. Under this NTK injectivity assumption, we prove that gradient flow converges to global minima when the initial loss is sufficiently small, eliminating spurious local minima from the optimization landscape.

Raphaël Barboni, Maarten V. de Hoop, Takashi Furuya +1

Bocconi University · Rice University · Doshisha University, RIKEN AIP +1

Transformer Wasserstein Gradient Flows Neural Network Training Dynamics Neural Tangent Kernel

May 17, 2026cs.LG

The Neural Tangent Kernel for Classification

In wide neural networks, the Neural Tangent Kernel (NTK) remains approximately constant during training, providing a powerful theoretical tool for studying training dynamics, generalization, and connections to kernel methods. However, this theory is largely restricted to regression losses. It was previously thought that training on a classification loss, or more generally losses involving nonlinear output transformations, breaks this property, leading to divergent logits and a breakdown of the linearization. In this paper, we extend NTK theory to classification by identifying conditions under which wide neural networks remain in the lazy training regime. We show that parameter-space regularization ensures a constant NTK during training for cross-entropy loss, while in the absence of regularization the regime is recovered when targets are non-degenerate, i.e. when all classes have strictly positive probability. Under these conditions, training is well-approximated by the linearized model, yielding an explicit characterization of the solution in terms of the NTK. We further analyze the distribution of trained predictors induced by random initialization and relate this notion of model uncertainty to Bayesian methods.

Jonathan Plenk, Sergio Calvo-Ordonez, Alvaro Cartea +3

Mathematical Institute, University of Oxford · Oxford-Man Institute of Quantitative Finance, University of Oxford · OATML, University of Oxford +2

Classification Neural Network Training Dynamics Neural Tangent Kernel

May 14, 2026physics.geo-ph

Deciphering Neural Reparameterized Full-Waveform Inversion with Neural Sensitivity Kernel and Wave Tangent Kernel

Full-waveform inversion (FWI) estimates unknown parameters in the wave equation from limited boundary measurements. Recent advances in neural reparameterized FWI (NeurFWI) demonstrate that representing the parameters using a neural network can reduce the reliance on the high-quality initial model and wavefield data, at the cost of slow high-resolution convergence. However, its underlying theoretical mechanism remains unclear. In this study, we establish the neural sensitivity kernel (NSK) and the wave tangent kernel (WTK) to analyze their convergence behavior from both model and data domains. These theoretical frameworks show that the neural tangent kernel (NTK) induced by neural representation adaptively modulates the original sensitivity and wave tangent kernels. This modulation leads to several key outcomes, i.e., the spectral filtering effect, the gradient wavenumber modulation, and the wave frequency bias, connecting the convergence behavior of NeurFWI with the eigen-structures of NSK and WTK. Building on these insights, we propose several enhanced NeurFWI methods with tailored eigen-structures in NSK and WTK to improve inversion performances and efficiency. We numerically validate these theoretical claims and the proposed methods in seismic exploration, and firstly extend their application to medical imaging.

Ruihua Chen, Yisi Luo, Bangyu Wu +2

School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an, Shaanxi 710049, China · School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an, Shaanxi, China · School of Mathematical Sciences, University of Electronic Science and Technology of China, Chengdu 610000, China

Neural Tangent Kernel Inverse Problems Image Inverse Problems

May 13, 2026cs.LG

Force-Aware Neural Tangent Kernels for Scalable and Robust Active Learning of MLIPs

Active learning for machine-learning interatomic potentials (MLIPs) must address several challenges to be practical: scaling to large candidate pools, leveraging energy-force supervision, and maintaining robustness when candidate pools are biased relative to the target distribution. In this work, we jointly address these challenges. We first introduce a linearly scaling acquisition framework based on chunked feature-space posterior-variance shortlisting. By avoiding materialisation of the candidate and train set kernels, this approach enables screening of ~200k structures within hours and applies broadly to acquisition strategies that score candidates based on molecular similarity metrics. We then extend the Neural Tangent Kernel (NTK) to a force-aware setting via mixed parameter-coordinate derivatives, yielding a force NTK and a joint energy-force NTK that provide natural similarity metrics for vector-field prediction. We demonstrate the effectiveness of the joint energy-force NTK on the OC20 dataset, where force-aware acquisition is crucial: it achieves the lowest energy and force MAE and RMSE across all metrics and distribution splits. Across T1x, PMechDB, and RGD benchmarks, our force NTK methods remain competitive with established baselines while being significantly more efficient than committee-based approaches. Under a controlled candidate-pool shift case study on T1x, acquisition based on pretrained MLIP embeddings and NTKs remains robust, whereas committee-based methods exhibit higher variance. Overall, these results show that a single pretrained MLIP can enable scalable, force-aware, and distribution-robust active learning for foundation-model fine-tuning.

Eszter Varga-Umbrich, Zachary Weller-Davies, Paul Duckworth +3

InstaDeep, UK · Department of Engineering, University of Cambridge, Cambridge, UK

Machine Learning Interatomic Potentials Distribution Shift Active Learning Neural Tangent Kernel

May 12, 2026cs.LG

State-Space NTK Collapse Near Bifurcations

Rich feature learning in tasks that unfold over time often requires the model to pass through bifurcations, constituting qualitative changes in the underlying model dynamics. We develop a local theory of gradient descent near these transitions through the empirical state-space neural tangent kernel (sNTK). Our central finding is that bifurcations both dominate and simplify learning dynamics: near bifurcations, we can reduce sNTK to a rank-one operator corresponding to learning in a classical normal form system, providing an analytically tractable description of the local learning geometry, even for high-dimensional recurrent systems. Concretely, we give a procedure for decomposing sNTK into bifurcation-relevant and residual channels, showing that near commonly codimension-1 bifurcations the relevant channel is a rank-one operator that is highly amplified. This amplification causes the bifurcation channel to dominate the full sNTK. Thus, bifurcations locally warp the learning landscape, funneling gradient descent into a few critical dynamical directions and making the nearby kernel and loss geometry predictable from classical normal forms. We illustrate this in a student-teacher recurrent neural network: the first learned bifurcation coincides with a sharp collapse in sNTK effective rank and the emergence of a dominant parameter direction whose restricted sNTK closely matches the landscape predicted by the scalar pitchfork normal form. Finally, we show that low-rank natural gradient methods resolve the resulting learning instability near bifurcations with very little overhead over SGD.

James Hazelden, Eric Shea-Brown

University of Washington

Recurrent Neural Networks Neural Network Training Dynamics Neural Tangent Kernel

May 9, 2026cs.LG

The Global Empirical NTK: Self-Referential Bias and Dimensionality of Gradient Descent Learning

In training a neural network with gradient descent (GD), each iteration induces a linear operator that governs first-order updates to a model's internal state variables. We define this operator as the Global Empirical Neural Tangent Kernel (NTK). In finite-width networks, the NTK is typically intractable to form, leading prior work to focus on restrictive settings such as tracking outputs only or taking infinite-width limits. Here, we study the structure of the NTK for a range of models. Formulating the model state as the solution to a single global implicit constraint, we derive the NTK as a product of two operators: K, accounting for immediate parameter-to-state interactions, and P, describing internal state-to-state dependencies. For a broad class of weight-based models, including RNNs and transformers, we prove a universal Kronecker-core theorem showing that K admits an exact, computable form given by the Gram matrix of weight-site variables. This core structure reveals that the NTK is structurally bottlenecked, constraining its effective rank and giving rise to a self-referential bias whereby GD preferentially learns within dominant modes of joint hidden and input activity. For recurrent models, we examine the spectrum of the NTK and show when it is biased and low-rank in space or time under the proposed decomposition. We further demonstrate that model dynamics at initialization bias the NTK, restricting learning and preventing task components from being learned effectively. Finally, we show that the NTK associated with a self-attention transformer is likewise structurally constrained to be low-rank. Overall, we show that the NTK possesses tractable structure that explains GD bias toward task solutions and the emergence of low-rank representations. To enable use of the NTK as a practical metric, we build kpflow, a library relying on randomized matrix-free numerical linear algebra.

James Hazelden, Laura Driscoll, Eli Shlizerman +1

Applied Mathematics, University of Washington; Allen Institute · Allen Institute · Neurobiology & Biophysics, University of Washington +2

Representation Learning Neural Tangent Kernel Gradient Descent Dynamics

May 8, 2026cs.LG

Convergence Analysis of Newton's Method for Neural Networks in the Overparameterized Limit

A convergence analysis is developed for the regularized Newton method for training neural networks (NNs) in the overparameterized limit. As the number of hidden units tends to infinity, the NN training dynamics converge in probability to the solution of a deterministic limit equation involving a ``Newton neural tangent kernel'' (NNTK). Explicit rates characterizing this convergence are provided and, in the infinite-width limit, we prove that the NN converges exponentially fast to the target data (i.e., a global minimizer with zero loss). We show that this convergence is uniform across the frequency spectrum, addressing the spectral bias inherent in gradient descent. The eigenvalues of the NTK for gradient descent accumulate at zero, leading to slow convergence for target data with high-frequency components. In contrast, the NNTK has uniformly lower bounded eigenvalues if the regularization parameter is selected appropriately, allowing Newton's method to converge more quickly for data with high-frequency components. Mathematical challenges that need to be addressed in our analysis include the implicit parameter update of the Newton method with a potentially indefinite Hessian matrix and the fact that the dimension of this linear system of equations tends to infinity as the NN width grows. This complicates deriving the training dynamics in the overparameterized limit as well as proving the convergence of the finite-width dynamics thereto. The analysis identifies a scaling formula for selecting the regularization parameter, which we show can vanish at a suitable rate as the number of hidden units becomes larger. We prove that, for sufficiently large numbers of hidden units, the regularized Hessian remains positive definite during training and the Newton updates for individual NN parameters converge to zero, showing that the model behaves as a linearization around the initialization.

Konstantin Riedl, Konstantinos Spiliopoulos, Justin Sirignano

Mathematical Institute University of Oxford · Department of Mathematics & Statistics Boston University

Neural Network Optimization Spectral Bias Second-Order Optimization Neural Network Training Dynamics Neural Tangent Kernel

May 8, 2026cond-mat.dis-nn

Spectral Dynamics in Deep Networks: Feature Learning, Outlier Escape, and Learning Rate Transfer

We study the evolution of hidden-weight spectra in wide neural networks trained by (stochastic) gradient descent. We develop a two-level dynamical mean-field theory (DMFT) that jointly tracks bulk and outlier spectral dynamics for spiked ensembles whose spike directions remain statistically dependent on the random bulk. We apply this framework to two settings: (1) infinite-width nonlinear networks in mean-field/

μ

P scaling and (2) deep linear networks in the proportional high-dimensional limit, where width, input dimension, and sample size diverge with fixed ratios. Our theory predicts how outliers evolve with training time, width, output scale, and initialization variance. In deep linear networks,

μ

P yields width-consistent outlier dynamics and hyperparameter transfer, including width-stable growth of the leading NTK mode toward the edge of stability (EoS). In contrast, NTK parameterization exhibits strongly width-dependent outlier dynamics, despite converging to a stable large-width limit. We show that this bulk+outlier picture is descriptive of simple tasks with small output channels, but that tasks involving large numbers of outputs (ImageNet classification or GPT language modeling) are better described by a restructuring of the spectral bulk. We develop a toy model with extensive output channels that recapitulates this phenomenon and show that edge of the spectrum still converges for sufficiently wide networks.

Clarissa Lauditi, Cengiz Pehlevan, Blake Bordelon

John A. Paulson School of Engineering and Applied Sciences, Harvard University · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University · Center for Mathematical Sciences and Applications, Harvard University +1

Deep Linear Networks Neural Network Training Dynamics Neural Tangent Kernel

May 5, 2026cs.LG

Pretrained Model Representations as Acquisition Signals for Active Learning of MLIPs

Training machine learning interatomic potentials (MLIPs) for reactive chemistry is often bottlenecked by the high cost of quantum chemical labels and the scarcity of transition state configurations in candidate pools. Active learning (AL) can mitigate these costs, but its effectiveness hinges on the acquisition rule. We investigate whether the latent space of a pretrained MLIP already contains the information necessary for effective acquisition, eliminating the need for auxiliary uncertainty heads, Bayesian training and fine-tuning, or committee ensembles. We introduce two acquisition signals derived directly from a pretrained MACE potential: a finite-width neural tangent kernel (NTK) and an activation kernel built from hidden latent space features. On reactive-chemistry benchmarks, both kernels consistently outperform fixed-descriptor baselines, committee disagreement, and random acquisition, reducing the data required to reach performance targets by an average of 38% for energy error and 28% for force error. We further show that the pretrained model induces similarity spaces that preserve chemically meaningful structure and provide more reliable residual uncertainty estimates than randomly initialised or fixed-descriptor-based kernels. Our results suggest that pretraining aligns latent-space geometry with model error, yielding a practical and sufficient acquisition signal for reactive MLIP fine-tuning.

Eszter Varga-Umbrich, Shikha Surana, Paul Duckworth +3

1InstaDeep · University of Cambridge

Machine Learning Interatomic Potentials Active Learning Kernel Methods Neural Tangent Kernel

May 5, 2026cs.LG

Rethinking the Rank Threshold for LoRA Fine-Tuning

A recent landscape analysis of LoRA fine-tuning in the neural tangent kernel regime establishes a sufficient condition

r(r+1)/2 > KN

on the LoRA rank

r

for the absence of spurious local minima under squared-error loss, prescribing

r \geq 12

on canonical few-shot RoBERTa setups. The condition is stated for general output dimension

K

, so its sharpness in any particular regime, and its practical implication for the cross-entropy loss actually used in fine-tuning, are open. We give three results that together reduce the prescribed rank to

r = 1

for binary classification in this regime. First, replacing the symmetric Sard-form count with the non-symmetric LoRA manifold dimension yields a strictly weaker capacity requirement,

r(m+n) - r^2 > C^* \cdot KN

with

C^* \approx 1.35

under Gaussian-iid features, satisfied at

r = 1

on canonical setups. Second, in the cross-entropy setting the Polyak--Łojasiewicz inequality removes the rank threshold entirely. Third, a Rademacher-complexity bound predicts rank-one variance optimality precisely when the bias term is saturated, which is the case for binary classification but not for

K > 2

. Empirically, across four GLUE-style binary tasks, three encoder architectures, and at scale on RoBERTa-large, rank one is competitive with the existing prescription

r = 12

; on multi-class MNLI the optimal rank shifts above one, also as predicted. The binary-regime guarantees are conditional on standard NTK assumptions; the multi-class extension is left to future work.

Juneyoung Park

OptAI Inc.

Low-Rank Adaptation Binary Classification Neural Tangent Kernel Parameter-Efficient Fine-Tuning

May 5, 2026cs.LG

Learning Dynamics of Zeroth-Order Optimization: A Kernel Perspective

Classical optimization theory establishes that zeroth-order (ZO) algorithms suffer from a dimension-dependent slowdown, with convergence rates typically scaling with the model dimension compared to first-order methods. However, in contrast to these theoretical expectations, a growing body of recent work demonstrates the successful application of ZO methods to fine-tuning Large Language Models (LLMs) with billions of parameters. To explain this paradox, we derive the one-step learning dynamics of ZO SGD, where the empirical Neural Tangent Kernel (eNTK) naturally emerges as the key term governing the learning behavior. Inspection of the eNTK produced by ZO SGD reveals that each element corresponds to the inner product of neural tangent vectors projected onto a random low-dimensional subspace. Thus, by invoking the Johnson-Lindenstrauss Lemma, our analysis shows that the fidelity of the ZO eNTK is governed primarily by the number of perturbations. Crucially, the approximation error depends on the model output size rather than the massive parameter dimension. This dimension-free property provides a theoretical justification for the scalability of ZO methods to LLMs finetuning tasks. We believe that this kernel-based framework offers a novel perspective for understanding ZO methods within the context of learning dynamics.

Zhe Li, Bicheng Ying, Zidong Liu +1

Department of Computing and Information Sciences Ph.D., Rochester Institute of Technology, Rochester, NY, USA · Google Inc., Los Angeles, CA, USA · ComboCurve Inc., Houston, TX, USA

Zeroth-Order Optimization LLM Fine-Tuning Neural Network Training Dynamics Neural Tangent Kernel

May 2, 2026cs.LG

A Theory of Generalization in Deep Learning

We present a non-asymptotic theory of generalization in deep learning where the empirical neural tangent kernel partitions the output space. In directions corresponding to signal, error dissipates rapidly; in the vast orthogonal dimensions corresponding to noise, the kernel's near-zero eigenvalues trap residual error in a test-invisible reservoir. Within the signal channel, minibatch SGD ensures that coherent population signal accumulates via fast linear drift, while idiosyncratic memorization is suppressed into a slow, diffusive random walk. We prove generalization survives even when the kernel evolves

\mathcal{O}(1)

in operator norm, the full feature-learning regime. This theory naturally explains disparate phenomena in deep learning theory, such as benign overfitting, double descent, implicit bias, and grokking. Lastly, we derive an exact population-risk objective from a single training run with no validation data, for any architecture, loss, or optimizer, and prove that it measures precisely the noise in the signal channel. This objective reduces in practice to an SNR preconditioner on top of Adam, adding one state vector at no extra cost; it accelerates grokking by

5 \times

, suppresses memorization in PINNs and implicit neural representations, and improves DPO fine-tuning under noisy preferences while staying

3 \times

closer to the reference policy.

Elon Litman, Gabe Guo

Stanford University

Neural Network Generalization Benign Overfitting Neural Network Training Dynamics Neural Tangent Kernel

May 1, 2026cs.LG

Topological Neural Tangent Kernel

Graph neural tangent kernels give a principled infinite-width theory for graph neural networks, but inherit a basic limitation of graph models: they see only pairwise structure. Many relational systems contain higher-order interactions that are more naturally represented by simplicial complexes. We introduce the Topological Neural Tangent Kernel (TopoNTK), an infinite-width kernel for simplicial message passing on edge features. TopoNTK combines lower Hodge interactions, capturing graph-like coupling through shared vertices, with upper Hodge interactions, capturing coupling through filled simplices. This makes the kernel sensitive to topology invisible to graph kernels, allowing complexes with the same graph but different filled simplices to induce different kernels. Beyond expressivity, the Hodge structure gives the kernel an interpretable learning geometry. Edge signals decompose into gradient-like, harmonic, and local circulation components, and the spectrum of the TopoNTK determines how quickly each component is learned. This yields a topological form of spectral bias: components aligned with large-eigenvalue modes are learned quickly, while global harmonic modes, retained through the residual channel, often lie at smaller eigenvalues and are learned more slowly. We prove expressivity, Hodge-alignment, spectral learning, and stability properties, and validate them on synthetic simplicial tasks and DBLP higher-order link prediction. The results show that topology is not merely extra structure; it can provide coordinates that make relational learning more faithful, interpretable, and effective.

Sanjukta Krishnagopal

Department of Computer Science, University of California Santa Barbara

Spectral Bias Neural Tangent Kernel Topological Representation Learning

Apr 28, 2026stat.ML

Adversarial Robustness of NTK Neural Networks

Deep learning models are widely deployed in safety-critical domains, but remain vulnerable to adversarial attacks. In this paper, we study the adversarial robustness of NTK neural networks in the context of nonparametric regression. We establish minimax optimal rates for adversarial regression in Sobolev spaces and then show that NTK neural networks, trained via gradient flow with early stopping, can achieve this optimal rate. However, in the overfitting regime, we prove that the minimum norm interpolant is vulnerable to adversarial perturbations.

Yuxuan Hou

1Qiuzhen College, Tsinghua University

Neural Network Robustness Adversarial Robustness Nonparametric Regression Neural Tangent Kernel

Apr 17, 2026cs.LG

Collective Kernel EFT for Pre-activation ResNets

In finite-width deep neural networks, the empirical kernel

G

evolves stochastically across layers. We develop a collective kernel effective field theory (EFT) for pre-activation ResNets based on a

G

-only closure hierarchy and diagnose its finite validity window. Exploiting the exact conditional Gaussianity of residual increments, we derive an exact stochastic recursion for

G

. Applying Gaussian approximations systematically yields a continuous-depth ODE system for the mean kernel

K_0

, the kernel covariance

V_4

, and the

1/n

mean correction

K_{1,\mathrm{EFT}}

, which emerges diagrammatically as a one-loop tadpole correction. Numerically,

K_0

remains accurate at all depths. However, the

V_4

equation residual accumulates to an

O(1)

error at finite time, primarily driven by approximation errors in the

G

-only transport term. Furthermore,

K_{1,\mathrm{EFT}}

fails due to the breakdown of the source closure, which exhibits a systematic mismatch even at initialization. These findings highlight the limitations of

G

-only state-space reduction and suggest extending the state space to incorporate the sigma-kernel.

Hidetoshi Kawase, Toshihiro Ota

CyberAgent, Inc., Shibuya, Tokyo 150–6121 Japan

Neural Tangent Kernel

Feb 4, 2026cs.LG

Gradient Flow Through Diagram Expansions: Learning Regimes and Explicit Solutions

We develop a general mathematical framework to analyze scaling regimes and derive explicit analytic solutions for gradient flow (GF) in large learning problems. Our key innovation is a formal power series expansion of the loss evolution, with coefficients encoded by diagrams akin to Feynman diagrams. We show that this expansion has a well-defined large-size limit that can be used to reveal different learning phases and, in some cases, to obtain explicit solutions of the nonlinear GF. We focus on learning Canonical Polyadic (CP) decompositions of high-order tensors, and show that this model has several distinct extreme lazy and rich GF regimes such as free evolution, NTK and under- and over-parameterized mean-field. We show that these regimes depend on the parameter scaling, tensor order, and symmetry of the model in a specific and subtle way. Moreover, we propose a general approach to summing the formal loss expansion by reducing it to a PDE; in a wide range of scenarios, it turns out to be first-order and solvable by the method of characteristics. We observe a very good agreement of our theoretical predictions with experimental results.

Dmitry Yarotsky, Eugene Golikov, Yaroslav Gusev

Applied AI Institute, Moscow, Russia · Steklov Mathematical Institute of Russian Academy of Sciences, Moscow, Russia.

Tensor Decomposition Mean-Field Theory Neural Tangent Kernel Gradient Descent Dynamics