cs.LGMay 9, 2026

The Global Empirical NTK: Self-Referential Bias and Dimensionality of Gradient Descent Learning

Authors: James Hazelden, Laura Driscoll, Eli Shlizerman, Eric Shea-Brown

Organizations: Applied Mathematics, University of Washington; Allen Institute · Allen Institute · Neurobiology & Biophysics, University of Washington · Applied Mathematics, Electrical & Computer Engineering, University of Washington · Applied Mathematics, Neurobiology & Biophysics, University of Washington; Allen Institute

Abstract

In training a neural network with gradient descent (GD), each iteration induces a linear operator that governs first-order updates to a model's internal state variables. We define this operator as the Global Empirical Neural Tangent Kernel (NTK). In finite-width networks, the NTK is typically intractable to form, leading prior work to focus on restrictive settings such as tracking outputs only or taking infinite-width limits. Here, we study the structure of the NTK for a range of models. Formulating the model state as the solution to a single global implicit constraint, we derive the NTK as a product of two operators: K, accounting for immediate parameter-to-state interactions, and P, describing internal state-to-state dependencies. For a broad class of weight-based models, including RNNs and transformers, we prove a universal Kronecker-core theorem showing that K admits an exact, computable form given by the Gram matrix of weight-site variables. This core structure reveals that the NTK is structurally bottlenecked, constraining its effective rank and giving rise to a self-referential bias whereby GD preferentially learns within dominant modes of joint hidden and input activity. For recurrent models, we examine the spectrum of the NTK and show when it is biased and low-rank in space or time under the proposed decomposition. We further demonstrate that model dynamics at initialization bias the NTK, restricting learning and preventing task components from being learned effectively. Finally, we show that the NTK associated with a self-attention transformer is likewise structurally constrained to be low-rank. Overall, we show that the NTK possesses tractable structure that explains GD bias toward task solutions and the emergence of low-rank representations. To enable use of the NTK as a practical metric, we build kpflow, a library relying on randomized matrix-free numerical linear algebra.

Explore similar work

CardsList
Jun 22, 2026cs.LG

Convergence of Gradient Descent for General Neural Network Architectures Beyond the NTK Regime

Training dynamics is central to understanding neural networks, yet its theoretical analysis remains difficult even for simple architectures and becomes substantially more challenging for general modern architectures. In this paper, we propose a convergence framework for analyzing gradient descent (GD) dynamics under a broad family of neural network architectures and datasets beyond the neural tangent kernel (NTK) regime. The framework is formulated at the level of network blocks and covers architectures including pre-normalized multi-layer transformers. More precisely, under mild assumptions, we prove that for almost all initializations, GD with regular learning rates converges to the neighbourhood of a stationary point. This is mainly proved by establishing an iterate-dependent PL-type inequality through analyticity and measure-zero arguments, and by proving Lipschitz smoothness along the GD trajectory through polynomial generalized smoothness and a local relaxed dissipative condition. We further interpret the theorem under Xavier initialization and practical architectural scaling, showing that the learning rate scale depends on the depth and effective bottleneck dimensions rather than the largest width. Finally, we derive structural nondegeneracy implications for residual connections and function composition, and provide a generic characterization of global minimizers within our framework.
Yuqing Wang
Neural Tangent KernelGradient DescentLearning RatesNeural NetworkNeural ArchitecturesConvergence
AskPDF
May 12, 2026cs.LG

State-Space NTK Collapse Near Bifurcations

Rich feature learning in tasks that unfold over time often requires the model to pass through bifurcations, constituting qualitative changes in the underlying model dynamics. We develop a local theory of gradient descent near these transitions through the empirical state-space neural tangent kernel (sNTK). Our central finding is that bifurcations both dominate and simplify learning dynamics: near bifurcations, we can reduce sNTK to a rank-one operator corresponding to learning in a classical normal form system, providing an analytically tractable description of the local learning geometry, even for high-dimensional recurrent systems. Concretely, we give a procedure for decomposing sNTK into bifurcation-relevant and residual channels, showing that near commonly codimension-1 bifurcations the relevant channel is a rank-one operator that is highly amplified. This amplification causes the bifurcation channel to dominate the full sNTK. Thus, bifurcations locally warp the learning landscape, funneling gradient descent into a few critical dynamical directions and making the nearby kernel and loss geometry predictable from classical normal forms. We illustrate this in a student-teacher recurrent neural network: the first learned bifurcation coincides with a sharp collapse in sNTK effective rank and the emergence of a dominant parameter direction whose restricted sNTK closely matches the landscape predicted by the scalar pitchfork normal form. Finally, we show that low-rank natural gradient methods resolve the resulting learning instability near bifurcations with very little overhead over SGD.
James Hazelden, Eric Shea-Brown
Neural Tangent KernelBifurcationGradient DescentDynamics Models
AskPDF
Jul 7, 2026stat.ML

A Function-Space Dichotomy for Compositional Learning: Exponential Sub-Optimality of the Neural Tangent Kernel

A persistent empirical observation is that trained neural networks outperform their neural tangent kernel (NTK) limit on tasks with compositional structure, yet a quantitative account of when\textbf{when} and by how much\textbf{by how much} has been lacking. Working on the unit circle, we give such an account through a dichotomy between two complexity measures of the target: its Fourier complexity\textbf{Fourier complexity}, which controls NTK kernel regression, and its architectural complexity\textbf{architectural complexity}, which controls learning over depth-LL, width-ww ReLU networks with the variation norm of the weights bounded by RR. We first characterize the minimax rate of the architecture class CL,w,R\mathcal{C}_{L,w,R}, pinning it down up to a single factor of LL: between Ω(Lw2R2/n)Ω(Lw^2R^2/n) and O~(L2w2R2/n)\tilde{O}(L^2w^2R^2/n). We then show the NTK estimator sits exponentially\textbf{exponentially} above this floor whenever the two complexities decouple: for the depth-LL iterated sawtooth, NTK regression needs Ω(4L)Ω(4^L) samples while the minimax floor is polynomial in LL. Numerical experiments confirm the theoretical claims: on bandlimited smooth targets, the NTK is competitive or better, while on the hypercube sparse-parity model, a standard two-layer network beats the NTK by four to six orders of magnitude in test error. The gap is thus a function-space property, a mismatch between the kernel's smoothness bias and the target's compositional structure, rather than a generic kernel-versus-network phenomenon.
Arkaprabha Ganguli, Emil Constantinescu
Neural Tangent KernelRectified Linear Unit NetworksWeight-SpaceSmoothingOptimality
AskPDF