Neural Tangent Kernel

The Neural Tangent Kernel (NTK) framework provides a powerful analytical tool for understanding the training dynamics of wide neural networks, essentially treating them as kernel methods in the infinite-width limit. Current research focuses on applying NTK to analyze various model architectures and training algorithms, including variational autoencoders, federated learning, and physics-informed neural networks, investigating issues like spectral bias, convergence rates, and generalization performance. This work offers valuable insights into the optimization and generalization properties of deep learning models, potentially leading to improved training strategies and a deeper understanding of their behavior.

Papers

January 3, 2024

Generalization Error Curves for Analytic Spectral Algorithms under Power-law Decay
Yicheng Li, Weiye Gan, Zuoqiang Shi, Qian Lin
Neural Tangent Kernel Generalization Error Kernel Regression Kernel Based Spectral Algorithm

January 2, 2024

Optimal Rates of Kernel Ridge Regression under Source Condition in Large Dimensions
Haobo Zhang, Yicheng Li, Weihao Lu, Qian Lin
Neural Tangent Kernel Data Dimensionality Kernel Ridge Regression Kernel Regression Optimal Rate High Dimensional Regime Source Filter

December 20, 2023

A note on regularised NTK dynamics with an application to PAC-Bayesian training
Eugenio Clerico, Benjamin Guedj
Neural Network Application Proficiency Regularization Model Neural Tangent Kernel Short Note Generalization Theory

December 6, 2023

An Infinite-Width Analysis on the Jacobian-Regularised Training of a Neural Network
Taeyoung Kim, Hongseok Yang
Neural Network Deep Neural Network Multi Layer Neural Tangent Kernel Hidden Layer Infinite Width Jacobian Regularization

December 4, 2023

Rethinking Adversarial Training with Neural Tangent Kernel
Guanlin Li, Han Qiu, Shangwei Guo, Jiwei Li, Tianwei Zhang
Adversarial Training Neural Tangent Kernel Training Dynamic Kernel Based Batch Normalization Layer

November 2, 2023

On the Convergence of Encoder-only Shallow Transformers
Yongtao Wu, Fanghui Liu, Grigorios G Chrysos, Volkan Cevher
Transformer Based Early Stage Convergence Neural Tangent Kernel Shallow Transformer

October 28, 2023

Efficient kernel surrogates for neural network-based regression
Saad Qadeer, Andrew Engel, Amanda Howard, Adam Tsou, Max Vargas, Panos Stinis, Tony Chiang
Deep Neural Network Neural Tangent Kernel Deep Regression DNN Accuracy Width Neural Network Kernel Based Surrogate

October 25, 2023

Unraveling Feature Extraction Mechanisms in Neural Networks
Xiaobing Sun, Jiaxi Li, Wei Lu
Large Language Model Neural Network CNN Model Feature Extraction Neural Tangent Kernel Language Modeling Task

October 21, 2023

On the Neural Tangent Kernel of Equilibrium Models
Zhili Feng, J. Zico Kolter
Neural Tangent Kernel Fully Connected Neural Network Deep Equilibrium Infinite Depth Equilibrium Model

October 19, 2023

Provable Guarantees for Neural Networks via Gradient Feature Learning
Zhenmei Shi, Junyi Wei, Yingyu Liang
Neural Network Gradient Descent Gradient Based Neural Tangent Kernel Feature Learning Provable Guarantee Learned Feature

October 12, 2023

Differentially Private Non-convex Learning for Multi-layer Neural Networks
Hanpu Shen, Cheng-Long Wang, Zihang Xiang, Yiming Ying, Di Wang
Neural Tangent Kernel Two Layer Neural Network Private Stochastic Multi Layer Neural Network

October 9, 2023

September 29, 2023

On the Disconnect Between Theory and Practice of Neural Networks: Limits of the NTK Perspective
Jonathan Wenger, Felix Dangel, Agustinus Kristiadi
Neural Network Large Scale Theoretical Understanding Continuum Limit Neural Tangent Kernel Kernel Regime Disengagement Report

September 26, 2023

September 14, 2023

September 9, 2023

Approximation Results for Gradient Descent trained Neural Networks
G. Welper
Neural Network Gradient Descent Gradient Flow Neural Tangent Kernel Smoothness Assumption Approximation Property

September 8, 2023

Connecting NTK and NNGP: A Unified Theoretical Framework for Neural Network Learning Dynamics in the Kernel Regime
Yehonatan Avidan, Qianyi Li, Haim Sompolinsky
Pre Trained Unified Framework Neural Tangent Kernel Training Dynamic Neural Network Gaussian Process Diffusion Learning Kernel Regime Time Dependent Neural