Neural Tangent Kernel

The Neural Tangent Kernel (NTK) framework provides a powerful analytical tool for understanding the training dynamics of wide neural networks, essentially treating them as kernel methods in the infinite-width limit. Current research focuses on applying NTK to analyze various model architectures and training algorithms, including variational autoencoders, federated learning, and physics-informed neural networks, investigating issues like spectral bias, convergence rates, and generalization performance. This work offers valuable insights into the optimization and generalization properties of deep learning models, potentially leading to improved training strategies and a deeper understanding of their behavior.

Papers

June 20, 2022

Limitations of the NTK for Understanding Generalization in Deep Learning
Nikhil Vyas, Yamini Bansal, Preetum Nakkiran
Deep Learning Strong Generalization Neural Tangent Kernel Fundamental Limitation Kernel Size Layer NTK Scaling Deep Network

June 17, 2022

June 13, 2022

Why Quantization Improves Generalization: NTK of Binary Weight Neural Networks
Kaiqi Zhang, Ming Yin, Yu-Xiang Wang
Neural Network Strong Generalization Quantization Operator Neural Tangent Kernel Reproducing Kernel Hilbert Space Gaussian Kernel Binary Weight

June 9, 2022

June 8, 2022

Identifying good directions to escape the NTK regime and efficiently learn low-degree plus sparse polynomials
Eshaan Nichani, Yu Bai, Jason D. Lee
Neural Tangent Kernel Interpretable Direction Polynomial System Sparse Polynomial

June 6, 2022

Spectral Bias Outside the Training Set for Deep Networks in the Kernel Regime
Benjamin Bowman, Guido Montufar
Deep Network Neural Tangent Kernel Fisher Information Spectral Bias Training Set Kernel Regime Width Network Quantitative Bound

June 4, 2022

On the Generalization Power of the Overfitted Three-Layer Neural Tangent Kernel Model
Peizhong Ju, Xiaojun Lin, Ness B. Shroff
Generalization Performance Neural Tangent Kernel Model Overfitting Hidden Layer Generalization Capacity Layer NTK

May 28, 2022

Tuning Frequency Bias in Neural Network Training with Nonuniform Data
Annan Yu, Yunan Yang, Alex Townsend
Neural Network Neural Tangent Kernel Generalization Error Non Uniform Frequency Bias

May 26, 2022

A Framework for Overparameterized Learning
Dávid Terjék, Diego González-Sánchez
New Framework Gradient Descent Multi Layer Neural Tangent Kernel Composite Optimization Implicit Regularization Effect

May 25, 2022

Analyzing Tree Architectures in Ensembles via Neural Tangent Kernel
Ryuichi Kanoh, Mahito Sugiyama
Decision Tree Ensemble Learning Diverse Ensemble Neural Tangent Kernel Oblique Decision Tree

May 24, 2022

May 20, 2022

Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterization
Simone Bombari, Mohammad Hossein Amani, Marco Mondelli
Deep Neural Network Optimization Purpose Neural Tangent Kernel Limited Memorization Generalization Guarantee N$ Neuron

April 30, 2022

NeuralEF: Deconstructing Kernels by Deep Neural Networks
Zhijie Deng, Jiaxin Shi, Jun Zhu
Deep Neural Network Neural Tangent Kernel Small Kernel Neural Network Gaussian Process Eigenfunction Decomposition

April 15, 2022

The training response law explains how deep neural networks learn
Kenichi Nakazato
Deep Neural Network Training Data Neural Tangent Kernel Mean Field Small Kernel

April 8, 2022

Single-level Adversarial Data Synthesis based on Neural Tangent Kernels
Yu-Rong Zhang, Ruei-Yang Su, Sheng Yen Chou, Shan-Hung Wu
Generative Model Generative Adversarial Network Adversarial Learning Neural Tangent Kernel Adversarial Pair

March 28, 2022

Demystifying the Neural Tangent Kernel from a Practical Perspective: Can it be trusted for Neural Architecture Search without training?
Jisoo Mok, Byunggook Na, Ji-Hoon Kim, Dongyoon Han, Sungroh Yoon
Training Data Neural Architecture Search Neural Architecture Neural Tangent Kernel Practical Approach Modern Neural Network Architecture

March 27, 2022

On the Neural Tangent Kernel Analysis of Randomly Pruned Neural Networks
Hongru Yang, Zhangyang Wang
Neural Network Neural Tangent Kernel Random Participation Fully Connected Neural Network Random Pruning