Deep Random Neural Network

Deep random neural networks, characterized by randomly initialized weights, are studied to understand the fundamental properties of neural networks and improve their training and performance. Current research focuses on analyzing the behavior of these networks in the infinite-width limit, exploring connections to Gaussian processes and Gaussian mixtures, and examining the impact of different weight distributions (e.g., sparse, low-rank) and normalization techniques (e.g., batch normalization) on their properties. This research provides valuable insights into the generalization ability and training dynamics of neural networks, potentially leading to more efficient training algorithms and improved model architectures for various applications, including intrusion detection and privacy-preserving data sharing.

Papers

May 16, 2024

Random ReLU Neural Networks as Non-Gaussian Processes
Rahul Parhi, Pakshal Bohra, Ayoub El Biari, Mehrsa Pourya, Michael Unser
Gaussian Process Shallow Neural Network Non Gaussian Random Neural Network Deep Random Neural Network

October 25, 2023

Beyond IID weights: sparse and low-rank deep Neural Networks are also Gaussian Processes
Thiziri Nait-Saada, Alireza Naderi, Jared Tanner
Gaussian Process Bayesian Neural Network Many Sparse Low Rank Wide Neural Network Initial Weight Deep Random Neural Network

September 4, 2023

Les Houches Lectures on Deep Learning at Large & Infinite Width
Yasaman Bahri, Boris Hanin, Antonin Brossollet, Vittorio Erba, Christian Keup, Rosalba Pacelli, James B. Simon
Deep Learning Deep Neural Network Statistical Mechanic Lecture Note Infinite Width Width Network Deep Random Neural Network

June 22, 2023

Online Self-Supervised Deep Learning for Intrusion Detection Systems
Mert Nakıp, Erol Gelenbe
Deep Learning Self Supervised Intrusion Detection System Based Anomaly Detection Online Deep Learning Deep Random Neural Network

March 31, 2023

PEOPL: Characterizing Privately Encoded Open Datasets with Public Labels
Homa Esfahanizadeh, Adam Yala, Rafael G. L. D'Oliveira, Andrea J. D. Jaba, Victor Quach, Ken R. Duffy, Tommi S. Jaakkola, Vinod Vaikuntanathan, Manya Ghobadi, Regina Barzilay, Muriel Médard
Data Encoding Manual Label Private Datasets Open Dataset Information Leakage Deep Random Neural Network

February 1, 2023

Bayes-optimal Learning of Deep Random Networks of Extensive-width
Hugo Cui, Florent Krzakala, Lenka Zdeborová
Neural Network Ridge Regression Kernel Regression Random Weight Bayes Optimal Deep Random Neural Network Bayes Error

May 25, 2022

On Bridging the Gap between Mean Field and Finite Width in Deep Random Neural Networks with Batch Normalization
Amir Joudaki, Hadi Daneshmand, Francis Bach
Neural Network Mean Field Batch Normalization Concentration Bound Deep Random Neural Network

April 26, 2022

Convergence of neural networks to Gaussian mixture distribution
Yasuhiko Asao, Ryotaro Sakamoto, Shiro Takagi
Neural Network Early Stage Convergence Gaussian Mixture Fully Connected Layer Output Deep Random Neural Network