Homogeneous Neural Network

Homogeneous neural networks, characterized by a scaling symmetry in their weight parameters, are a focus of current research aiming to understand the optimization dynamics and implicit biases of deep learning models. Studies are concentrating on the impact of initialization strategies, particularly the "Goldilocks zone" of optimal initial weight norms, and the resulting gradient flow dynamics near saddle points and minima. This research sheds light on phenomena like "grokking" and the surprising predictability of out-of-distribution extrapolations, offering insights into training stability, generalization, and robustness. These findings have implications for improving training efficiency, enhancing model interpretability, and developing more reliable deep learning systems.

Papers

November 5, 2024

A Convex Relaxation Approach to Generalization Analysis for Parallel Positively Homogeneous Networks
Uday Kiran Reddy Tadipatri, Benjamin D. Haeffele, Joshua Agterberg, René Vidal
Empirical Risk Minimization Generalization Bound Convex Relaxation Homogeneous Neural Network Two Layer Network Homogeneous Diffusive Network

October 29, 2024

Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networks
Nikolaos Tsilivis, Gal Vardi, Julia Kempe
Neural Network Gradient Descent Implicit Bias Margin Maximization Homogeneity Bias Systematic Bias Homogeneous Neural Network

March 12, 2024

Early Directional Convergence in Deep Homogeneous Neural Networks for Small Initializations
Akshay Kumar, Jarvis Haupt
Homogeneous Neural Network Small Initialization Lipschitz Gradient Gradient Flow Dynamic

February 14, 2024

Directional Convergence Near Small Initializations and Saddles in Two-Homogeneous Neural Networks
Akshay Kumar, Jarvis Haupt
Neural Network Homogeneous Neural Network Gradient Flow Dynamic Saddle to Saddle Dynamic

February 5, 2024

Deconstructing the Goldilocks Zone of Neural Network Initialization
Artem Vysogorets, Anna Dawid, Julia Kempe
Pre Trained Hessian Matrix Gaussian Curvature Homogeneous Neural Network Neural Network Initialization SWEET Spot

November 30, 2023

Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokking
Kaifeng Lyu, Jikai Jin, Zhiyuan Li, Simon S. Du, Jason D. Lee, Wei Hu
Implicit Bias Regression Task Margin Classifier Grokking Phenomenon Homogeneous Neural Network

November 29, 2023

Homogeneous Artificial Neural Network
Andrey Polyakov
Neural Network Universal Approximation Homogeneity Bias Homogeneous Neural Network Weighted Sum Scalarization

October 12, 2023

A Survey of Heterogeneous Transfer Learning
Runxue Bao, Yiming Sun, Yuhe Gao, Jindong Wang, Qiang Yang, Haifeng Chen, Zhi-Hong Mao, Ye Ye
Timely Survey Transfer Learning Homogeneous Neural Network Heterogeneous Transfer Learning

October 2, 2023

Deep Neural Networks Tend To Extrapolate Predictably
Katie Kang, Amrith Setlur, Claire Tomlin, Sergey Levine
Deep Neural Network Loss Function Distribution Data Dimensional Input Neural Network Prediction Homogeneous Neural Network

May 26, 2023

A Hybrid Neural Coding Approach for Pattern Recognition with Spiking Neural Networks
Xinyi Chen, Qu Yang, Jibin Wu, Haizhou Li, Kay Chen Tan
Neural Network Spiking Neural Network Pattern Recognition Homogeneous Neural Network Hybrid Encoder

February 2, 2023

Understanding Reconstruction Attacks with the Neural Tangent Kernel and Dataset Distillation
Noel Loo, Ramin Hasani, Mathias Lechner, Alexander Amini, Daniela Rus
Neural Tangent Kernel Dataset Distillation Reconstruction Attack Data Reconstruction Attack Reconstructed Image Homogeneous Neural Network

January 18, 2023

Catapult Dynamics and Phase Transitions in Quadratic Nets
David Meltzer, Junyu Liu
Learning Rate Phase Transition Wide Neural Network Quadratic Neural Network Homogeneous Neural Network

October 18, 2022

Enabling Heterogeneous Domain Adaptation in Multi-inhabitants Smart Home Activity Learning
Md Mahmudur Rahman, Mahta Mousavi, Peri Tarr, Mohammad Arif Ul Alam
Domain Adaptation Activity Recognition Semi Supervised Domain Adaptation Sensor Based Human Activity Recognition Homogeneous Neural Network Heterogeneous Domain Adaptation

October 13, 2022

Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Data
Spencer Frei, Gal Vardi, Peter L. Bartlett, Nathan Srebro, Wei Hu
Gradient Descent Low Rank Gradient Flow High Dimensional Data Implicit Bias Fully Connected Neural Network Homogeneous Neural Network Leaky ReLU

October 7, 2022

The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networks
Daniel Kunin, Atsushi Yamamura, Chao Ma, Surya Ganguli
Gradient Flow Homogeneous Neural Network Advanced Activation Mechanism Gradient Dynamic Maximization Bias

May 31, 2022

Feature Learning in $L_{2}$-regularized DNNs: Attraction/Repulsion and Sparsity
Arthur Jacot, Eugene Golikov, Clément Hongler, Franck Gabriel
Sparsity Increase DNN Framework Feature Learning Layer Wise Hidden Representation External Human Force Homogeneous Neural Network L2 Regularization N$ Neuron

April 9, 2022

Neuronal diversity can improve machine learning for physics and beyond
Anshul Choudhary, Anil Radhakrishnan, John F. Lindner, Sudeshna Sinha, William L. Ditto
Neural Network Machine Learning Theoretical Physic Hamiltonian Neural Network Homogeneous Neural Network Conventional Neural Network Neuronal Diversity

December 30, 2021

Are we really making much progress? Revisiting, benchmarking, and refining heterogeneous graph neural networks
Qingsong Lv, Ming Ding, Qiang Liu, Yuxiang Chen, Wenzheng Feng, Siming He, Chang Zhou, Jianguo Jiang, Yuxiao Dong, Jie Tang
Neural Network Heterogeneous Graph Much Progress Benchmark Score Reproducible Research Homogeneous Neural Network

November 15, 2021

Fast Axiomatic Attribution for Neural Networks
Robin Hesse, Simone Schaub-Meyer, Stefan Roth
Neural Network Attribution Score Homogeneous Neural Network Attribution Evaluation Attribution Model