Three Layer Neural Network

Three-layer neural networks are a fundamental building block in deep learning, currently under intense investigation to understand their training dynamics, feature learning capabilities, and generalization performance. Research focuses on analyzing the impact of network architecture (e.g., width, activation functions), data properties (e.g., dimensionality, signal-to-noise ratio), and training algorithms (e.g., gradient descent) on network behavior, often employing models like MLP-Mixers and variations of restricted Boltzmann machines. These studies aim to provide theoretical guarantees for their performance and shed light on the effectiveness of different training strategies, ultimately improving the design and application of these networks in various fields.

Papers

November 12, 2024

Exploring the loss landscape of regularized neural networks via convex duality
Sungyoon Kim, Aaron Mishkin, Mert Pilanci
Neural Network Loss Landscape Two Layer Neural Network Vector Valued Strong Duality Three Layer Neural Network

September 3, 2024

Beyond Unconstrained Features: Neural Collapse for Shallow Neural Networks with General Data
Wanli Hong, Shuyang Ling
Deep Neural Network Raw Data Class Imbalance Neural Collapse Shallow Neural Network Three Layer Neural Network Unconstrained Feature Model

June 18, 2024

Hierarchical Associative Memory, Parallelized MLP-Mixer, and Symmetry Breaking
Ryo Karakida, Toshihiro Ota, Masato Taki
Single Neural Network MLP Mixer Symmetry Breaking Three Layer Neural Network

May 22, 2024

Interpolation with deep neural networks with non-polynomial activations: necessary and sufficient numbers of neurons
Liam Madden
Neural Network Polynomial Activation Three Layer Neural Network Memory Capacity Input Output Pair N$ Neuron

May 19, 2024

Error Analysis of Three-Layer Neural Network Trained with PGD for Deep Ritz Method
Yuling Jiao, Yanming Lai, Yang Wang
Gradient Descent Partial Differential Equation Error Analysis Three Layer Neural Network Deep Ritz Method

February 25, 2024

On the dynamics of three-layer neural networks: initial condensation
Zheng-An Chen, Tao Luo
Neural Network Two Layer Neural Network Three Layer Neural Network Initial Condensation

November 23, 2023

Learning Hierarchical Polynomials with Three-Layer Neural Networks
Zihao Wang, Eshaan Nichani, Jason D. Lee
O$ Sample Three Layer Neural Network

November 15, 2023

SparseSpikformer: A Co-Design Framework for Token and Weight Pruning in Spiking Transformer
Yue Liu, Shanlin Xiao, Bo Li, Zhiyi Yu
Spiking Neural Network K TOKEN Co Design Spiking Transformer Weight Pruning Three Layer Neural Network Spikformer Architecture

October 9, 2023

Investigating Continuous Learning in Spiking Neural Networks
C. Tanner Fredieu
Neural Network Spiking Neural Network Neural Network Architecture Continuous Learning SNN Model Three Layer Neural Network

September 30, 2023

Memorization with neural nets: going beyond the worst case
Sjoerd Dirksen, Patrick Finke, Martin Genzel
Neural Network Neural Network Architecture Limited Memorization Interpolation Regime Worst Case Three Layer Neural Network Memorization Capacity

May 26, 2023

Generalization Guarantees of Gradient Descent for Multi-Layer Neural Networks
Puyu Wang, Yunwen Lei, Di Wang, Yiming Ying, Ding-Xuan Zhou
Neural Network Gradient Descent Two Layer Neural Network Generalization Guarantee Multi Layer Neural Network Three Layer Neural Network

May 25, 2023

Data Topology-Dependent Upper Bounds of Neural Network Widths
Sangmin Lee, Jong Chul Ye
Deep Neural Network Raw Data Universal Approximation Upper Bound Layer ReLU Network Three Layer Neural Network Width Neural Network

May 11, 2023

Provable Guarantees for Nonlinear Feature Learning in Three-Layer Neural Networks
Eshaan Nichani, Alex Damian, Jason D. Lee
Deep Learning Deep Network Layer Wise Provable Guarantee Three Layer Neural Network

April 14, 2023

A Bibliometric Review of Neuromorphic Computing and Spiking Neural Networks
Nicholas J. Pritchard, Andreas Wicenec, Mohammed Bennamoun, Richard Dodson
Spiking Neural Network Neuromorphic Computing Neuromorphic Hardware Low Power Computational Power Bibliometric Analysis Three Layer Neural Network

March 16, 2023

Bayesian Generalization Error in Linear Neural Networks with Concept Bottleneck Structure and Multitask Formulation
Naoki Hayashi, Yoshihide Sawada
Multi Task Generalization Performance Generalization Error Concept Bottleneck Model Linear Neural Network Three Layer Neural Network

October 28, 2022

A Functional-Space Mean-Field Theory of Partially-Trained Three-Layer Neural Networks
Zhengdao Chen, Eric Vanden-Eijnden, Joan Bruna
Mean Field Two Layer Neural Network Three Layer Neural Network Kernel Flow

October 27, 2022

May 24, 2022

Empirical Phase Diagram for Three-layer Neural Networks with Infinite Width
Hanxu Zhou, Qixuan Zhou, Zhenyuan Jin, Tao Luo, Yaoyu Zhang, Zhi-Qin John Xu
Gradient Flow Deeper Network Two Layer ReLU Rectified Linear Unit Three Layer Neural Network Phase Diagram Infinite Width Dynamical Regime

April 13, 2022

Deep Learning Model with GA based Feature Selection and Context Integration
Ranju Mandal, Basim Azam, Brijesh Verma, Mengjie Zhang
Deep Learning Model Feature Selection Image Segmentation CNN Model Genetic Algorithm Three Layer Neural Network