Simple Neural Network

Simple neural networks, despite their seemingly basic architecture, are a subject of ongoing research focused on understanding their surprising effectiveness and exploring their limitations. Current investigations delve into representing their nonlinear operations using linear algebra techniques, developing efficient training methods for specialized hardware like optical networks, and analyzing their generalization capabilities to mitigate issues like hallucinations. This research is significant because it enhances our fundamental understanding of neural network behavior, improves training efficiency and resource utilization, and contributes to the development of more robust and reliable machine learning models for various applications.

Papers

October 18, 2024

Topological obstruction to the training of shallow ReLU neural networks
Marco Nurisso, Pierrick Leroy, Francesco Vaccarino
Training Data Gradient Flow ReLU Activation Neural Scaling Law Shallow ReLU Simple Neural Network

October 10, 2024

Generalization from Starvation: Hints of Universality in LLM Knowledge Graph Learning
David D. Baek, Yuxiao Li, Max Tegmark
Strong Generalization Inherent Interpretability Graph Learning K$ Universality Natural Language Hint LLM in Context Simple Neural Network

September 2, 2024

Representing Neural Network Layers as Linear Operations via Koopman Operator Theory
Nishant Suresh Aswani, Saif Eddin Jabari, Muhammad Shafique
Dynamical System Koopman Operator Dynamic Mode Decomposition Nonlinear Activation Neural Network Layer Simple Neural Network

August 15, 2024

Training Large-Scale Optical Neural Networks with Two-Pass Forward Propagation
Amirreza Ahmadnejad, Somayyeh Koohi
Neural Network Convolutional Neural Network Training Data Backward Propagation Optical Neural Network Optical Computing Simple Neural Network

June 25, 2024

Banishing LLM Hallucinations Requires Rethinking Generalization
Johnny Li, Saksham Consul, Eda Zhou, James Wong, Naila Farooqui, Yuxin Ye, Nithyashree Manohar, Zhuxiaona Wei, Tian Wu, Ben Echols, Sharon Zhou, Gregory Diamos
Systematic Generalization Type II Hallucination LLM Hallucination Simple Neural Network

July 3, 2023

May 9, 2023

Novel Synthetic Data Tool for Data-Driven Cardboard Box Localization
Lukáš Gajdošech, Peter Kravár
Neural Network Synthetic Data Data Generation Procedural Model Simple Neural Network

January 6, 2023

Grokking modular arithmetic
Andrey Gromov
Gradient Descent Vanilla Gradient Descent Simple Neural Network Modular Addition

November 28, 2022

Weight Predictor Network with Feature Selection for Small Sample Tabular Biomedical Data
Andrei Margeloiu, Nikola Simidjievski, Pietro Lio, Mateja Jamnik
Feature Selection Tabular Data Biomedical Datasets Classification Network Simple Neural Network Weight Estimation

November 7, 2022

Automatic Change-Point Detection in Time Series via Deep Learning
Jie Li, Paul Fearnhead, Piotr Fryzlewicz, Tengyao Wang
Deep Learning Time Series Change Point Detection Accelerometer Data Change Point Simple Neural Network

October 4, 2022

Low-Light Image Restoration Based on Retina Model using Neural Networks
Yurui Ming, Yuanyuan Liang
Neural Network Neural Network Model Low Light Visual Circuit Simple Neural Network Retina Model Optical Neuron

May 9, 2022

Statistical Guarantees for Approximate Stationary Points of Simple Neural Networks
Mahsa Taheri, Fang Xie, Johannes Lederer
Neural Network Statistical Model Statistical Guarantee Deep Learning Pipeline Simple Neural Network Stationary Point

April 21, 2022

Accelerating Machine Learning via the Weber-Fechner Law
B. N. Kausik
Machine Learning Human Perception Simple Neural Network Human Concept

March 15, 2022

Reachability In Simple Neural Networks
Marco Sälzer, Martin Lange
Neural Network Lower Bound Reachability Analysis Neural Network Verification NP Complete Reachability Based Safety Analysis Simple Neural Network

December 3, 2021

NN-LUT: Neural Approximation of Non-Linear Operations for Efficient Transformer Inference
Joonsang Yu, Junki Park, Seongmin Park, Minsoo Kim, Sihwa Lee, Dong Hyun Lee, Jungwook Choi
Transformer Model Transformer Inference Neural Network Approximation Simple Neural Network Hardware Approximation Nonlinear Operation