Empirical Neural Tangent Kernel

The empirical neural tangent kernel (eNTK) approximates the behavior of neural networks during training, offering a valuable tool for understanding their generalization performance and implicit regularization. Current research focuses on improving eNTK approximations for efficiency and accuracy, exploring its relationship to phenomena like neural collapse and leveraging it for tasks such as data selection and federated learning. This work is significant because it bridges the theory-practice gap in deep learning, providing insights into model behavior and enabling the development of more robust and efficient training methods. The eNTK's application in explainable AI and improved federated learning algorithms highlights its practical impact.

Papers

October 9, 2024

Collective variables of neural networks: empirical time evolution and scaling laws
Samuel Tovey, Sven Krippendorf, Michael Spannowsky, Konstantin Nikolaou, Christian Holm
Neural Network Deep Learning Architecture Neural Network Representation Temporal Evolution Collective Variable Regime Switching Empirical Neural Tangent Kernel

January 16, 2024

lpNTK: Better Generalisation with Less Data via Sample Interaction During Learning
Shangmin Guo, Yi Ren, Stefano V. Albrecht, Kenny Smith
Neural Network LeArning Abstract Strong Generalization Generalization Performance Neural Tangent Kernel Better Generalization Less Data Interaction Understanding Empirical Neural Tangent Kernel

November 13, 2023

A PAC-Bayesian Perspective on the Interpolating Information Criterion
Liam Hodgkinson, Chris van der Heide, Robert Salomone, Fred Roosta, Michael W. Mahoney
Generalization Performance Implicit Regularization PAC Bayesian Bias Variance Information Criterion Empirical Neural Tangent Kernel

May 25, 2023

Neural (Tangent Kernel) Collapse
Mariia Seleznova, Dana Weitzner, Raja Giryes, Gitta Kutyniok, Hung-Hsu Chou
Neural Tangent Kernel Neural Topic Neural Collapse Event Collapse Empirical Neural Tangent Kernel

May 23, 2023

Faithful and Efficient Explanations for Neural Networks via Neural Tangent Kernel Surrogate Models
Andrew Engel, Zhichao Wang, Natalie S. Frank, Ioana Dumitriu, Sutanay Choudhury, Anand Sarwate, Tony Chiang
Neural Network Neural Tangent Kernel Kernel Function Effective Explanation Empirical Neural Tangent Kernel

May 1, 2023

Towards a Phenomenological Understanding of Neural Networks: Data
Samuel Tovey, Sven Krippendorf, Konstantin Nikolaou, Christian Holm
Neural Network Raw Data Data Selection Collective Variable Empirical Neural Tangent Kernel Random Network Distillation

July 13, 2022

TCT: Convexifying Federated Learning using Bootstrapped Neural Tangent Kernels
Yaodong Yu, Alexander Wei, Sai Praneeth Karimireddy, Yi Ma, Michael I. Jordan
Optimization Purpose Empirical Neural Tangent Kernel

June 25, 2022

A Fast, Well-Founded Approximation to the Empirical Neural Tangent Kernel
Mohamad Amin Mohamadi, Wonho Bae, Danica J. Sutherland
Kernel Matrix Approximation Theory Empirical Neural Tangent Kernel O$ Memory

May 24, 2022

TorchNTK: A Library for Calculation of Neural Tangent Kernels of PyTorch Models
Andrew Engel, Zhichao Wang, Anand D. Sarwate, Sutanay Choudhury, Tony Chiang
Convolutional Neural Network Multi Layer Neural Tangent Kernel Neural Network Model Pytorch Model Easy to Use Library Different Calculation Method Empirical Neural Tangent Kernel

December 3, 2021

A Structured Dictionary Perspective on Implicit Neural Representations
Gizem Yüce, Guillermo Ortiz-Jiménez, Beril Besbinar, Pascal Frossard
Signal Representation Deep Learning Theory Dictionary Based Empirical Neural Tangent Kernel