Neural Tangent Kernel

The Neural Tangent Kernel (NTK) framework provides a powerful analytical tool for understanding the training dynamics of wide neural networks, essentially treating them as kernel methods in the infinite-width limit. Current research focuses on applying NTK to analyze various model architectures and training algorithms, including variational autoencoders, federated learning, and physics-informed neural networks, investigating issues like spectral bias, convergence rates, and generalization performance. This work offers valuable insights into the optimization and generalization properties of deep learning models, potentially leading to improved training strategies and a deeper understanding of their behavior.

Papers

March 17, 2022

March 14, 2022

March 10, 2022

Transition to Linearity of Wide Neural Networks is an Emerging Property of Assembling Weak Models
Chaoyue Liu, Libin Zhu, Mikhail Belkin
Neural Network Neural Tangent Kernel Wide Neural Network Late Time Transition Weak Model

February 27, 2022

The Spectral Bias of Polynomial Neural Networks
Moulik Choraria, Leello Tadesse Dadi, Grigorios Chrysos, Julien Mairal, Volkan Cevher
Neural Network Neural Tangent Kernel Spectral Bias Polynomial Neural Network Frequency Function

February 13, 2022

A Geometric Understanding of Natural Gradient
Qinxun Bai, Steven Rosenberg, Wei Xu
Natural Gradient Neural Tangent Kernel Geometric Understanding

February 10, 2022

Deep Learning in Random Neural Fields: Numerical Experiments via Neural Tangent Kernel
Kaito Watanabe, Kotaro Sakamoto, Ryo Karakida, Sho Sonoda, Shun-ichi Amari
Deep Learning Neural Field Neural Tangent Kernel Receptive Field Numerical Experiment

February 1, 2022

Neural Tangent Kernel Beyond the Infinite-Width Limit: Effects of Depth and Initialization
Mariia Seleznova, Gitta Kutyniok
Mixed Effect Neural Tangent Kernel Large Depth ReLU Network New Initialization Infinite Width Infinite Depth and Width Limit

January 28, 2022

Interplay between depth of neural networks and locality of target functions
Takashi Mori, Masahito Ueda
Neural Network Deep Neural Network Neural Tangent Kernel Large Depth Simple Function Wide Neural Network Generalization Capability

January 20, 2022

Kernel Methods and Multi-layer Perceptrons Learn Linear Models in High Dimensions
Mojtaba Sahraee-Ardakan, Melikasadat Emami, Parthe Pandit, Sundeep Rangan, Alyson K. Fletcher
High Dimensional Multi Layer Neural Tangent Kernel High Dimension Kernel Method Linear Model Kernel Based High Dimensional Regime

January 12, 2022

Implicit Bias of MSE Gradient Optimization in Underparameterized Neural Networks
Benjamin Bowman, Guido Montufar
Gradient Flow Neural Tangent Kernel Implicit Bias Gradient Based Optimization Eigenfunction Decomposition

December 31, 2021

How Infinitely Wide Neural Networks Can Benefit from Multi-task Learning -- an Exact Macroscopic Characterization
Jakob Heiss, Josef Teichmann, Hanna Wutte
Deep Neural Network Multi Task Learning Neural Tangent Kernel Wide Neural Network Macroscopic Model

December 15, 2021

Rethinking Influence Functions of Neural Networks in the Over-parameterized Regime
Rui Zhang, Shihua Zhang
Neural Network Black Box Neural Tangent Kernel Influence Function Regularization Loss Hessian Vector Product Parameterized Regime

December 7, 2021

Understanding Square Loss in Training Overparametrized Neural Network Classifiers
Tianyang Hu, Jun Wang, Wenjia Wang, Zhenguo Li
Deep Learning Training Data Loss Function Cross Entropy Loss Neural Tangent Kernel Neural Network Classifier Square Loss

December 4, 2021

Fast Graph Neural Tangent Kernel via Kronecker Sketching
Shunhua Jiang, Yunze Man, Zhao Song, Zheng Yu, Danyang Zhuo
Graph Neural Network Neural Tangent Kernel Kernel Matrix Kronecker Product Graph Neural Tangent Kernel

November 28, 2021

Neural Tangent Kernel of Matrix Product States: Convergence and Applications
Erdong Guo, David Draper
Financial Application Early Stage Convergence Neural Tangent Kernel Gaussian Kernel Kernel Function Matrix Product State

November 11, 2021

On the Equivalence between Neural Network and Support Vector Machine
Yilan Chen, Wei Huang, Lam M. Nguyen, Tsui-Wei Weng
Neural Network Support Vector Machine Neural Tangent Kernel Wide Neural Network Width Neural Network Infinite Width Neural Network

November 6, 2021

Understanding Layer-wise Contributions in Deep Neural Networks through Spectral Analysis
Yatin Dandi, Arthur Jacot
Deep Neural Network Neural Tangent Kernel Layer Wise Spectral Bias Spectral Analysis Frequency Function

Neural Tangent Kernel

Papers

A Framework and Benchmark for Deep Batch Active Learning for Regression

On the Spectral Bias of Convolutional Neural Tangent and Gaussian Process Kernels

On Connecting Deep Trigonometric Networks with Deep Gaussian Processes: Covariance, Expressivity, and Neural Tangent Kernel

Phenomenology of Double Descent in Finite-Width Neural Networks

Transition to Linearity of Wide Neural Networks is an Emerging Property of Assembling Weak Models

The Spectral Bias of Polynomial Neural Networks

A Geometric Understanding of Natural Gradient

Deep Learning in Random Neural Fields: Numerical Experiments via Neural Tangent Kernel

Neural Tangent Kernel Beyond the Infinite-Width Limit: Effects of Depth and Initialization

Interplay between depth of neural networks and locality of target functions

Kernel Methods and Multi-layer Perceptrons Learn Linear Models in High Dimensions

Implicit Bias of MSE Gradient Optimization in Underparameterized Neural Networks

How Infinitely Wide Neural Networks Can Benefit from Multi-task Learning -- an Exact Macroscopic Characterization

Rethinking Influence Functions of Neural Networks in the Over-parameterized Regime

Understanding Square Loss in Training Overparametrized Neural Network Classifiers

Fast Graph Neural Tangent Kernel via Kronecker Sketching

Neural Tangent Kernel of Matrix Product States: Convergence and Applications

On the Equivalence between Neural Network and Support Vector Machine

Understanding Layer-wise Contributions in Deep Neural Networks through Spectral Analysis