Neural Tangent Kernel

The Neural Tangent Kernel (NTK) framework provides a powerful analytical tool for understanding the training dynamics of wide neural networks, essentially treating them as kernel methods in the infinite-width limit. Current research focuses on applying NTK to analyze various model architectures and training algorithms, including variational autoencoders, federated learning, and physics-informed neural networks, investigating issues like spectral bias, convergence rates, and generalization performance. This work offers valuable insights into the optimization and generalization properties of deep learning models, potentially leading to improved training strategies and a deeper understanding of their behavior.

Papers

April 6, 2023

NTK-SAP: Improving neural network pruning by aligning training dynamics
Yite Wang, Dawei Li, Ruoyu Sun
Neural Network Neural Tangent Kernel Training Dynamic

March 26, 2023

Analyzing Convergence in Quantum Neural Networks: Deviations from Neural Tangent Kernels
Xuchen You, Shouvanik Chakrabarti, Boyang Chen, Xiaodi Wu
Neural Network Early Stage Convergence Neural Tangent Kernel Quantum Neural Network Intermediate Scale Quantum Subject Level Deviation Quantum Version

March 7, 2023

Benign Overfitting for Two-layer ReLU Convolutional Neural Networks
Yiwen Kou, Zixiang Chen, Yuanzhou Chen, Quanquan Gu
Neural Tangent Kernel Two Layer ReLU Benign Overfitting

February 12, 2023

Generalization Ability of Wide Neural Networks on $\mathbb{R}$
Jianfa Lai, Manyun Xu, Rui Chen, Qian Lin
Neural Tangent Kernel Wide Neural Network Two Layer ReLU Benign Overfitting Neural Kernel

February 3, 2023

Beyond the Universal Law of Robustness: Sharper Laws for Random Features and Neural Tangent Kernels
Simone Bombari, Shayan Kiyani, Marco Mondelli
Native Robustness Adversarial Perturbation Empirical Risk Minimization Neural Tangent Kernel Random Feature Benign Over Parameterization Universal Law

February 2, 2023

February 1, 2023

Gradient Descent in Neural Networks as Sequential Learning in RKBS
Alistair Shilton, Sunil Gupta, Santu Rana, Svetha Venkatesh
Neural Network Gradient Descent Neural Tangent Kernel Kernel Hilbert Space Sequential Learning Generalization Property Taylor Expansion Weak Convergence Kernel Banach Space

January 26, 2023

A Simple Algorithm For Scaling Up Kernel Methods
Teng Andrea Xu, Bryan Kelly, Semyon Malamud
Neural Tangent Kernel Kernel Method Random Feature Wide Neural Network Random Feature Regression Simple Algorithm

January 14, 2023

Understanding the Spectral Bias of Coordinate Based MLPs Via Training Dynamics
John Lazzari, Xiuwen Liu
Neural Tangent Kernel Training Dynamic Spectral Bias Coordinate Based Frequency Representation

January 7, 2023

Investigations on convergence behaviour of Physics Informed Neural Networks across spectral ranges and derivative orders
Mayank Deshpande, Siddharth Agarwal, Vukka Snigdha, Arya Kumar Bhattacharya
Loss Function Physic Informed Neural Network Neural Tangent Kernel Comprehensive Investigation Spectral Bias Convergence Behavior Derivative Process Spectral Band

January 1, 2023

Neural Networks with Sparse Activation Induced by Large Bias: Tighter Analysis with Bias-Generalized NTK
Hongru Yang, Ziyu Jiang, Ruizhe Zhang, Yingbin Liang, Zhangyang Wang
Strong Generalization Early Stage Convergence Neural Tangent Kernel ReLU Network Sparse Network Wide Neural Network Sparse Activation Relative Stance Bias

December 28, 2022

Learning Lipschitz Functions by GD-trained Shallow Overparameterized ReLU Neural Networks
Ilja Kuzborskij, Csaba Szepesvári
Gradient Descent Neural Tangent Kernel Shallow ReLU Lipschitz Operator Width Neural Network Lipschitz Learning

November 26, 2022

Simple initialization and parametrization of sinusoidal networks via their kernel bandwidth
Filipe de Avila Belbute-Peres, J. Zico Kolter
Neural Tangent Kernel Implicit Model Kernel Extension Sinusoidal Activation Structured Initialization

November 15, 2022

Characterizing the Spectrum of the NTK via a Power Series Expansion
Michael Murray, Hui Jin, Benjamin Bowman, Guido Montufar
Neural Network Neural Tangent Kernel Time to Spectrum Taylor Expansion Hermite Interpolation

November 11, 2022

Spectral Evolution and Invariance in Linear-width Neural Networks
Zhichao Wang, Andrew Engel, Anand Sarwate, Ioana Dumitriu, Tony Chiang
Learning Rate Neural Tangent Kernel Spectral Feature Alpha Invariance Conventional Neural Network

November 7, 2022

Can neural networks extrapolate? Discussion of a theorem by Pedro Domingos
Adrien Courtois, Jean-Michel Morel, Pablo Arias
Neural Network Gradient Descent Neural Tangent Kernel Discussion Paper Theorem Driven Question Kernel Machine

October 23, 2022

Accelerated Linearized Laplace Approximation for Bayesian Deep Learning
Zhijie Deng, Feng Zhou, Jun Zhu
Bayesian Neural Network Neural Tangent Kernel Bayesian Deep Learning Gauss Newton Laplace Approximation

October 21, 2022

Evolution of Neural Tangent Kernels under Benign and Adversarial Training
Noel Loo, Ramin Hasani, Alexander Amini, Daniela Rus
Adversarial Attack Adversarial Training Adversarial Robustness Neural Tangent Kernel Specie Evolution Adversarial Vulnerability Benign Input

October 18, 2022

Disentangling the Predictive Variance of Deep Ensembles through the Neural Tangent Kernel
Seijin Kobayashi, Pau Vilimelis Aceituno, Johannes von Oswald
Inductive Bias Neural Tangent Kernel Deep Ensemble Input Uncertainty