Random Weight

Random weight initialization in neural networks explores the surprising effectiveness of using untrained or partially trained networks with randomly assigned weights for various machine learning tasks. Current research focuses on understanding the theoretical properties of such networks, including their approximation capabilities and convergence rates, often employing models like Random Vector Functional Link networks and analyzing architectures with ReLU activation functions. This research is significant because it challenges conventional training paradigms, potentially leading to more efficient algorithms and a deeper understanding of neural network generalization, with applications ranging from time series clustering to federated learning.

Papers

November 7, 2024

Finding Strong Lottery Ticket Networks with Genetic Algorithms
Philipp Altmann, Julian Schönberger, Maximilian Zorn, Thomas Gabor
Neural Network Genetic Algorithm Lottery Ticket Lottery Ticket Hypothesis Random Weight

August 15, 2024

RandomNet: Clustering Time Series Using Untrained Deep Neural Networks
Xiaosheng Li, Wenjie Xi, Jessica Lin
Neural Network Deep Network Time Series Clustering Random Weight UCR Time Series

July 1, 2024

Expressivity of Neural Networks with Random Weights and Learned Biases
Ezekiel Williams, Avery Hee-Woon Ryoo, Thomas Jiralerspong, Alexandre Payeur, Matthew G. Perich, Luca Mazzucato, Guillaume Lajoie
Neural Network Universal Approximation Behavior Expressivity Style Synaptic Weight Random Weight

May 27, 2024

Large Deviations of Gaussian Neural Networks with ReLU activation
Quirin Vogel
Deep Neural Network ReLU Activation Random Weight Large Deviation Gaussian Neural Network

November 7, 2023

MixtureGrowth: Growing Neural Networks by Recombining Learned Parameters
Chau Pham, Piotr Teterwak, Soren Nelson, Bryan A. Plummer
Neural Network Deep Neural Network Mixture Component ImageNet Dataset Parameter Learning Random Weight Model Recombination

July 12, 2023

Quantitative CLTs in Deep Neural Networks
Stefano Favaro, Boris Hanin, Domenico Marinucci, Ivan Nourdin, Giovanni Peccati
Deep Neural Network High Dimensional Deeper Network Random Weight Quantitative Bound

June 30, 2023

Efficient uniform approximation using Random Vector Functional Link networks
Palina Salanevich, Olov Schavemaker
Hidden Layer ReLU Activation Random Weight Random Vector Functional Link Uniform Approximation

June 26, 2023

Chance-Constrained Multiple-Choice Knapsack Problem: Model, Algorithms, and Applications
Xuanfeng Li, Shengcai Liu, Jin Wang, Xiao Chen, Yew-Soon Ong, Ke Tang
Full Model Financial Application Practical Algorithm Combinatorial Optimization Knapsack Problem Random Weight Chance Constraint

May 29, 2023

A Rainbow in Deep Network Black Boxes
Florentin Guth, Brice Ménard, Gaspar Rochette, Stéphane Mallat
Black Box Dimensional Embeddings Q Network Random Weight Gaussian Bayesian Network

March 1, 2023

Learning curves for deep structured Gaussian feature models
Jacob A. Zavatone-Veth, Cengiz Pehlevan
Random Feature Deep Learning Theory Random Weight Learning Curve

February 1, 2023

Bayes-optimal Learning of Deep Random Networks of Extensive-width
Hugo Cui, Florent Krzakala, Lenka Zdeborová
Neural Network Ridge Regression Kernel Regression Random Weight Bayes Optimal Deep Random Neural Network Bayes Error

September 30, 2022

Sparse Random Networks for Communication-Efficient Federated Learning
Berivan Isik, Francesco Pase, Deniz Gunduz, Tsachy Weissman, Michele Zorzi
Sparse Network Gradient Compression Dense Network Random Network Random Weight

June 12, 2022

Universality and approximation bounds for echo state networks with random weights
Zhen Li, Yunfei Yang
Activation Function Echo State Network K$ Universality Approximation Guarantee Non Linear Operator Random Weight

June 9, 2022

Early Transferability of Adversarial Examples in Deep Neural Networks
Oriel BenShmuel
Deep Neural Network Adversarial Example Adversarial Perturbation Task Transferability Plausible Explanation Random Weight Adversarial Planning

June 2, 2022

A Bhattacharyya Coefficient-Based Framework for Noise Model-Aware Random Walker Image Segmentation
Dominik Drees, Florian Eilers, Ang Bian, Xiaoyi Jiang
New Framework Image Segmentation Segmentation Performance Random Walk Segmentation Method Interactive Image Segmentation Random Weight Class Separability

May 17, 2022

Deep neural networks with dependent weights: Gaussian Process mixture limit, heavy tails, sparsity and compressibility
Hoil Lee, Fadhel Ayed, Paul Jung, Juho Lee, Hongseok Yang, François Caron
Deep Neural Network Sparsity Increase Non Gaussian Random Weight Infinite Width Neural Network Heavy Tail Infinite Width

April 14, 2022

Concentration of Random Feature Matrices in High-Dimensions
Zhijun Chen, Hayden Schaeffer, Rachel Ward
High Dimension Random Feature Selective Focus Random Weight Random Feature Regression Nonlinear Matrix Decomposition

April 8, 2022

Learning to Modulate Random Weights: Neuromodulation-inspired Neural Networks For Efficient Continual Learning
Jinyung Hong, Theodore P. Pavlic
LeArning Abstract Continual LEArning Catastrophic Forgetting Efficient Continual Learning Random Weight Synaptic Dynamic

March 22, 2022

Learning curves for the multi-class teacher-student perceptron
Elisabetta Cornacchia, Francesca Mignacco, Rodrigo Veiga, Cédric Gerbelot, Bruno Loureiro, Lenka Zdeborová
High Dimensional Teacher Network Random Weight Learning Curve

January 12, 2022

Eigenvalue Distribution of Large Random Matrices Arising in Deep Neural Networks: Orthogonal Case
Leonid Pastur
Deep Neural Network Random Matrix Post Hoc Orthogonalization Large Matrix Jacobian Matrix Random Weight Singular Value Eigenvalue Distribution