Neural Tangent Kernel

The Neural Tangent Kernel (NTK) framework provides a powerful analytical tool for understanding the training dynamics of wide neural networks, essentially treating them as kernel methods in the infinite-width limit. Current research focuses on applying NTK to analyze various model architectures and training algorithms, including variational autoencoders, federated learning, and physics-informed neural networks, investigating issues like spectral bias, convergence rates, and generalization performance. This work offers valuable insights into the optimization and generalization properties of deep learning models, potentially leading to improved training strategies and a deeper understanding of their behavior.

Papers

October 18, 2022

Analysis of Convolutions, Non-linearity and Depth in Graph Neural Networks using Neural Tangent Kernel
Mahalakshmi Sabanayagam, Pascal Esser, Debarghya Ghoshdastidar
Graph Neural Network Direct Convolution Neural Tangent Kernel Graph Convolution Large Depth Non Linearity Graph Neural Tangent Kernel

October 13, 2022

Sample-Then-Optimize Batch Neural Thompson Sampling
Zhongxiang Dai, Yao Shu, Bryan Kian Hsiang Low, Patrick Jaillet
Bayesian Optimization Surrogate Model Neural Tangent Kernel Black Box Optimization Neural Thompson Sampling

October 12, 2022

Few-shot Backdoor Attacks via Neural Tangent Kernels
Jonathan Hayase, Sewoong Oh
Deep Neural Network Backdoor Attack Neural Tangent Kernel Poisoning Attack Attack Success Rate Shot Backdoor

October 11, 2022

October 10, 2022

September 29, 2022

Restricted Strong Convexity of Deep Learning Models with Smooth Activations
Arindam Banerjee, Pedro Cisneros-Velarde, Libin Zhu, Mikhail Belkin
Deep Learning Model Gradient Descent Neural Tangent Kernel Strong Convexity Smooth Activation Function

September 16, 2022

Extrapolation and Spectral Bias of Neural Nets with Hadamard Product: a Polynomial Net Study
Yongtao Wu, Zhenyu Zhu, Fanghui Liu, Grigorios G Chrysos, Volkan Cevher
Neural Network Neural Tangent Kernel Neural Network Architecture Extrapolation Framework Spectral Bias Standard Deep Polynomial Neural Network Hadamard Layer Polynomial Network

September 15, 2022

Generalization Properties of NAS under Activation and Skip Connection Search
Zhenyu Zhu, Fanghui Liu, Grigorios G Chrysos, Volkan Cevher
Neural Architecture Search Neural Architecture Neural Tangent Kernel Generalization Property Targeted Activation Penalty Generalization Error Bound Deep Learning Theory Skip Connection

September 9, 2022

Fast Neural Kernel Embeddings for General Activations
Insu Han, Amir Zandieh, Jaehoon Lee, Roman Novak, Lechao Xiao, Amin Karbasi
Neural Tangent Kernel Kernel Method Targeted Activation Penalty Neural Kernel

August 29, 2022

Neural Tangent Kernel: A Survey
Eugene Golikov, Eduard Pokonechnyy, Vladimir Korviakov
Neural Network Timely Survey Neural Tangent Kernel Kernel Method

August 19, 2022

Graph Convolutional Networks from the Perspective of Sheaves and the Neural Tangent Kernel
Thomas Gebhart
Convolutional Neural Network Visual Perspective Neural Tangent Kernel Deep Learning Algorithm Sheaf Diffusion Topological Generalization

August 7, 2022

An Empirical Analysis of the Laplace and Neural Tangent Kernels
Ronaldas Paulius Lencevicius
Neural Tangent Kernel Reproducing Kernel Hilbert Space Empirical Analysis Small Kernel Kernel Function Laplacian Kernel Laplace Explainer

July 14, 2022

July 7, 2022

Neural Stein critics with staged $L^2$-regularization
Matthew Repasky, Xiuyuan Cheng, Yao Xie
Regularization Model Neural Tangent Kernel L2 Regularization Weight Regularization

June 29, 2022

Momentum Diminishes the Effect of Spectral Bias in Physics-Informed Neural Networks
Ghazal Farhani, Alexander Kazachek, Boyu Wang
Gradient Descent Physic Informed Neural Network Mixed Effect Neural Tangent Kernel Adam Optimizer Residual Momentum Spectral Bias

June 26, 2022

Bounding the Width of Neural Networks via Coupled Initialization -- A Worst Case Analysis
Alexander Munteanu, Simon Omlor, Zhao Song, David P. Woodruff
Neural Network Neural Tangent Kernel Worst Case General Bound Two Layer ReLU Minimum Width Robust Initialization

June 25, 2022

Making Look-Ahead Active Learning Strategies Feasible with Neural Tangent Kernels
Mohamad Amin Mohamadi, Wonho Bae, Danica J. Sutherland
Active Learning Deep Network Neural Tangent Kernel Sequential Learning Pool Based Active Learning