Neural Network Gaussian Process

Neural Network Gaussian Processes (NNGPs) provide a theoretical framework for understanding the behavior of infinitely wide neural networks by representing their output as a Gaussian process. Current research focuses on refining NNGP kernels, comparing them to related kernels like the Neural Tangent Kernel (NTK), and leveraging them for tasks such as Bayesian inference, multiple imputation of missing data, and improved uncertainty quantification in deep learning models. This work bridges the gap between the theoretical analysis of neural networks and their practical application, offering insights into generalization, representation learning, and efficient training strategies.

Papers

June 4, 2024

Kernel vs. Kernel: Exploring How the Data Structure Affects Neural Collapse
Vignesh Kothapalli, Tom Tirer
Neural Tangent Kernel Kernel Method Neural Collapse Kernel Extension Data Structure Neural Network Gaussian Process

May 9, 2024

Wilsonian Renormalization of Neural Network Gaussian Processes
Jessica N. Howard, Ro Jefferson, Anindita Maiti, Zohar Ringel
Gaussian Process Flow Mood Renormalization Group Neural Network Gaussian Process

April 26, 2024

Multi-layer random features and the approximation power of neural networks
Rustem Takhanov
Neural Network Multi Layer Neural Architecture Kernel Hilbert Space Random Feature Random Field Approximation Power Neural Network Gaussian Process

March 26, 2024

A Unified Kernel for Neural Network Learning
Shao-Qun Zhang, Zong-Yi Chen, Yong-Ming Tian, Xun Lu
Neural Network Neural Tangent Kernel Neural Network Gaussian Process

February 9, 2024

Flexible infinite-width graph convolutional networks and the importance of representation learning
Ben Anson, Edward Milsom, Laurence Aitchison
Convolutional Neural Network Graph Neural Network Representation Learning Importance Aware Width Neural Network Neural Network Gaussian Process Flexible Graph

September 18, 2023

Convolutional Deep Kernel Machines
Edward Milsom, Ben Anson, Laurence Aitchison
Deep Gaussian Process Deep Kernel Learning Neural Network Gaussian Process

September 8, 2023

Connecting NTK and NNGP: A Unified Theoretical Framework for Neural Network Learning Dynamics in the Kernel Regime
Yehonatan Avidan, Qianyi Li, Haim Sompolinsky
Pre Trained Unified Framework Neural Tangent Kernel Training Dynamic Neural Network Gaussian Process Diffusion Learning Kernel Regime Time Dependent Neural

July 31, 2023

A theory of data variability in Neural Network Bayesian inference
Javed Lindner, David Dahmen, Michael Krämer, Moritz Helias
Theoretical Understanding Kernel Method Generalization Property Kernel Matrix Non Pathological Variability Neural Network Gaussian Process Bayesian Neural

May 17, 2023

Sparsity-depth Tradeoff in Infinitely Wide Deep Neural Networks
Chanwoo Chun, Daniel D. Lee
Sparse Network Wide Neural Network Sparse Activation Neural Network Gaussian Process Deep Bayesian

February 19, 2023

Guided Deep Kernel Learning
Idan Achituve, Gal Chechik, Ethan Fetaya
Deep Kernel Learning Deep Kernel Neural Network Gaussian Process Infinite Width Neural Network Kernel Optimization

November 23, 2022

Multiple Imputation with Neural Network Gaussian Process for High-dimensional Incomplete Data
Zongyu Dai, Zhiqi Bu, Qi Long
Imputation Method Neural Network Gaussian Process

October 12, 2022

Fast Bayesian Updates for Deep Learning with a Use Case in Active Learning
Marek Herde, Zhixin Huang, Denis Huseljic, Daniel Kottke, Stephan Vogt, Bernhard Sick
Deep Learning Active Learning Bayesian Neural Network Use Case Neural Network Gaussian Process Bayesian Update

June 15, 2022

Wide Bayesian neural networks have a simple weight posterior: theory and accelerated sampling
Jiri Hron, Roman Novak, Jeffrey Pennington, Jascha Sohl-Dickstein
Theoretical Understanding Bayesian Neural Network Markov Chain Monte Carlo Wide Neural Network Reparameterization Method Neural Network Gaussian Process Gaussian Posterior

May 1, 2022

A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awareness
Jeremiah Zhe Liu, Shreyas Padhy, Jie Ren, Zi Lin, Yeming Wen, Ghassen Jerfel, Zack Nado, Jasper Snoek, Dustin Tran, Balaji Lakshminarayanan
Uncertainty Quantification Simple Approach Neural Network Gaussian Process Probabilistic Deep Learning Deep Learning Model Uncertainty

April 30, 2022

NeuralEF: Deconstructing Kernels by Deep Neural Networks
Zhijie Deng, Jiaxin Shi, Jun Zhu
Deep Neural Network Neural Tangent Kernel Small Kernel Neural Network Gaussian Process Eigenfunction Decomposition

January 12, 2022

On neural network kernels and the storage capacity problem
Jacob A. Zavatone-Veth, Cengiz Pehlevan
Neural Network Two Layer Neural Network Wide Neural Network Neural Network Gaussian Process Neural Kernel