Neural Tangent Kernel

The Neural Tangent Kernel (NTK) framework provides a powerful analytical tool for understanding the training dynamics of wide neural networks, essentially treating them as kernel methods in the infinite-width limit. Current research focuses on applying NTK to analyze various model architectures and training algorithms, including variational autoencoders, federated learning, and physics-informed neural networks, investigating issues like spectral bias, convergence rates, and generalization performance. This work offers valuable insights into the optimization and generalization properties of deep learning models, potentially leading to improved training strategies and a deeper understanding of their behavior.

Papers

September 8, 2023

Optimal Rate of Kernel Regression in Large Dimensions
Weihao Lu, Haobo Zhang, Yicheng Li, Manyun Xu, Qian Lin
Neural Tangent Kernel High Dimensional Data Data Dimensionality Kernel Regression Optimal Rate

September 5, 2023

Representation Learning Dynamics of Self-Supervised Models
Pascal Esser, Satyaki Mukherjee, Debarghya Ghoshdastidar
Self Supervised Learning Contrastive Loss Supervised Learning Neural Tangent Kernel Learning Dynamic Self Supervised Model Dynamic Representation

August 31, 2023

On the Equivalence between Implicit and Explicit Neural Networks: A High-dimensional Viewpoint
Zenan Ling, Zhenyu Liao, Robert C. Qiu
Neural Tangent Kernel High Dimension Shallow Network Implicit Language Implicit Neural Network Explicit Feature Map

August 1, 2023

An Exact Kernel Equivalence for Finite Classification Models
Brian Bell, Michael Geyer, David Glickenstein, Amanda Fernandez, Juston Moore
Neural Tangent Kernel Kernel Method Transition Kernel Computation Kernel Parametric Learning

July 26, 2023

Controlling the Inductive Bias of Wide Neural Networks by Modifying the Kernel's Spectrum
Amnon Geifman, Daniel Barzilai, Ronen Basri, Meirav Galun
Gradient Descent Inductive Bias Neural Tangent Kernel Wide Neural Network Kernel Extension Time to Spectrum

July 25, 2023

Modify Training Directions in Function Space to Reduce Generalization Error
Yi Yu, Wenlian Lu, Boyu Chen
Neural Tangent Kernel Generalization Error Function Space Natural Gradient Descent

July 18, 2023

NTK-approximating MLP Fusion for Efficient Language Model Fine-tuning
Tianxin Wei, Zeming Guo, Yifan Chen, Jingrui He
Neural Network Neural Tangent Kernel

June 9, 2023

Efficient Uncertainty Quantification and Reduction for Over-Parameterized Neural Networks
Ziyi Huang, Henry Lam, Haofeng Zhang
Neural Network Uncertainty Quantification Neural Tangent Kernel Deep Ensemble Constructive Reduction High Uncertainty Value

June 7, 2023

NTKCPL: Active Learning on Top of Self-Supervised Model by Estimating True Coverage
Ziting Wen, Oscar Pizarro, Stefan Williams
Self Supervised Active Learning Estimation Task Neural Tangent Kernel Self Supervised Model Higher Coverage Rate ActIve Learning algORithm Novel Active Learning

June 6, 2023

Stochastic Marginal Likelihood Gradients using Neural Tangent Kernels
Alexander Immer, Tycho F. A. van der Ouderaa, Mark van der Wilk, Gunnar Rätsch, Bernhard Schölkopf
Hyperparameter Optimization Neural Tangent Kernel Related Hyperparameters Hyperparameter Selection Marginal Likelihood Bayesian Model Selection

May 30, 2023

Benign Overfitting in Deep Neural Networks under Lazy Training
Zhenyu Zhu, Fanghui Liu, Grigorios G Chrysos, Francesco Locatello, Volkan Cevher
Neural Network Deep Neural Network Neural Tangent Kernel Generalization Error Benign Overfitting Smooth Activation Function Lazy Training

May 25, 2023

Neural (Tangent Kernel) Collapse
Mariia Seleznova, Dana Weitzner, Raja Giryes, Gitta Kutyniok, Hung-Hsu Chou
Neural Tangent Kernel Neural Topic Neural Collapse Event Collapse Empirical Neural Tangent Kernel

May 23, 2023

May 22, 2023

May 20, 2023

How Spurious Features Are Memorized: Precise Analysis for Random and NTK Features
Simone Bombari, Marco Mondelli
Deep Learning Model Neural Tangent Kernel Generalization Error Feature Alignment Comprehensive Analysis Random Participation Spurious Feature Spurious Pattern

May 15, 2023

ReLU soothes the NTK condition number and accelerates optimization for wide neural networks
Chaoyue Liu, Like Hui
Optimization Purpose Activation Function Neural Tangent Kernel Wide Neural Network Linear Neural Network Rectified Linear Unit

May 4, 2023

On the Eigenvalue Decay Rates of a Class of Neural-Network Related Kernel Functions Defined on General Domains
Yicheng Li, Zixiong Yu, Guhan Chen, Qian Lin
Neural Tangent Kernel Open Domain True Class Wide Neural Network Kernel Function Eigenvalue Decay

April 19, 2023

Leveraging the two timescale regime to demonstrate convergence of neural networks
Pierre Marion, Raphaël Berthier
Neural Network Stochastic Gradient Descent Early Stage Convergence Gradient Flow Neural Tangent Kernel Shallow Neural Network Timescale Analysis Two Timescale

Neural Tangent Kernel

Papers

Optimal Rate of Kernel Regression in Large Dimensions

Representation Learning Dynamics of Self-Supervised Models

On the Equivalence between Implicit and Explicit Neural Networks: A High-dimensional Viewpoint

An Exact Kernel Equivalence for Finite Classification Models

Controlling the Inductive Bias of Wide Neural Networks by Modifying the Kernel's Spectrum

Modify Training Directions in Function Space to Reduce Generalization Error

NTK-approximating MLP Fusion for Efficient Language Model Fine-tuning

Efficient Uncertainty Quantification and Reduction for Over-Parameterized Neural Networks

NTKCPL: Active Learning on Top of Self-Supervised Model by Estimating True Coverage

Stochastic Marginal Likelihood Gradients using Neural Tangent Kernels

Benign Overfitting in Deep Neural Networks under Lazy Training

Neural (Tangent Kernel) Collapse

Faithful and Efficient Explanations for Neural Networks via Neural Tangent Kernel Surrogate Models

Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimension

Tight conditions for when the NTK approximation is valid

Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models

How Spurious Features Are Memorized: Precise Analysis for Random and NTK Features

ReLU soothes the NTK condition number and accelerates optimization for wide neural networks

On the Eigenvalue Decay Rates of a Class of Neural-Network Related Kernel Functions Defined on General Domains

Leveraging the two timescale regime to demonstrate convergence of neural networks