Shallow Neural Networks

Ask about this topic

Overview

Shallow neural networks—typically models with one hidden layer—are studied for what functions they can represent, how they learn, and when their compact structure is useful. Approximation theory uses tools such as Fourier and spherical-harmonic analysis to give error and generalization guarantees, while optimization research examines gradient descent, loss landscapes, symmetry, and mean-field dynamics. Practical approaches include changing activation functions, regularizing parameter norms, and making individual units more expressive, with applications ranging from differential equations to compact classifiers. A central tension is that width and richer units can improve approximation, but depth may be essential for some tasks, and representational capacity alone does not ensure that training finds a good solution.

Latest papers

CardsList

57 papers11 papers in the last four weeks, up 120% on the four weeks before. 0.1% of all new papers.

  1. Boundary Geometry Determines Optimal Shallow ReLU Approximation under Weighted Coefficient Constraints

    Oct 27, 2026Jia-Qi Yang, Lei ShiNeural Network Approximation TheoryShallow Neural Networks

    School of Mathematical Sciences and Shanghai Key Laboratory for Contemporary Applied Mathematics, Fudan University, Shanghai 200433, China.

  2. The Geometry of Limits in ReLU Function Spaces

    Oct 24, 2026Kathryn A. LindseyReLU Neural NetworksNeural Network Approximation Theory

  3. Silent Alignment in Two-Layer Linear Networks: A High-Dimensional Perspective

    Oct 21, 2026Simon Martin, Francis Bach, Giulio BiroliTwo-Layer Neural NetworksNeural Network Training Dynamics

    DI-ENS, LPENS, SIERRA · DI-ENS, SIERRA · LPENS

  4. Two ReLU⁡\operatorname{ReLU} Layers is All You Need

    Oct 20, 2026Florestan Brunck, Christoph Hertrich, Kilian Ruess +2ReLU Neural NetworksShallow Neural Networks

  5. Diffusion Score Estimation with Shallow Neural Networks: Regularity, Approximation, and Finite-Sample Rates

    Oct 19, 2026Fan Liu, Yahong Yang, Wei ZhuScore-Based Generative ModelingDiffusion Models

    School of Mathematics, Georgia Institute of Technology, USA · Department of Mathematics and Statistics, Binghamton University, USA

  6. Two hidden layer suffice: the caterpillar net

    Oct 19, 2026Jakob Paul ZimmermannReLU Neural NetworksShallow Neural Networks

    FU Berlin · TU Berlin

  7. A "feature ODE" describing the learning behavior of shallow MLPs on simple functions

    Oct 19, 2026Joseph Turnbull, Berkan Ottlik, Jamie SimonNeural Network Training DynamicsShallow Neural Networks

    UC Berkeley · University of Pennsylvania · Imbue

  8. Why Shallow Neural Networks Learn Sinusoidal Components First: An Orthogonal Polynomial Dynamics Framework

    Oct 15, 2026Yanyang XiaoSpectral BiasNeural Network Training Dynamics

    Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences

  9. Activation-Adjacent Layer Normalization: Effects on Approximation Power

    Oct 15, 2026Yunhao Ni, Yuxin Guo, Jie Luo +2Layer NormalizationNeural Network Approximation Theory

    State Key Laboratory of Complex and Critical Software Environment, Beihang University, Beijing, China · Hangzhou International Innovation Institute, Beihang University, Hangzhou, China

  10. Fluctuations of Nonlinear Observables in Mean Field Neural Network Training

    Oct 7, 2026Arnaud Descours, Geoffrey LacourNeural Network Training DynamicsUncertainty Quantification

    UCBL · MaIAGE

  11. Removing spurious minima for planar features by skip connections

    Oct 1, 2026Jakob Paul Zimmermann, Andrei Balakin, Moritz Grillo +1Shallow Neural NetworksReLU Neural Networks

    Freie Universität Berlin · Technische Universität Berlin

  12. The Conflict Between Logic and Memory: Training Conditions for Optimizer-Dependent Rule Acquisition

    Sep 30, 2026Gongyue Zhang, Honghai LiuShallow Neural NetworksMultilayer Perceptrons

  13. Population loss in shallow ReLU networks: Bias & families of critical points

    Sep 25, 2026Michael FieldShallow Neural NetworksReLU Neural Networks

  14. Error bounds in Sobolev norms for approximations with norm constrained ReLU neural networks

    Sep 17, 2026Xianjun Li, Yunfei YangShallow Neural NetworksReLU Neural Networks

    School of Statistics and Data Science, Capital University of Economics and Business, Beijing, China. · School of Mathematics (Zhuhai) and Guangdong Province Key Laboratory of Computational Science, Sun Yat-sen University, Zhuhai, China.

  15. Algebraic Expressivity Certificates for Shallow Polynomial Neural Networks

    Sep 11, 2026Sepehr Akbari, Shahrzad JamshidiNeural Network ExpressivityShallow Neural Networks

    Dept. of Math & Computer Science Lake Forest College

  16. Learning Orthogonal Multi-Index Models Beyond Small Initialization: Incremental Learning, Competitive Dynamics and Symmetry

    Sep 9, 2026Mo Zhou, Weihang Xu, Simon S. Du +1Shallow Neural NetworksOrthogonal Representation Learning

    University of Washington · Amazon, Inc.

  17. Shallow neural network approximation in mixed Sobolev spaces

    Sep 4, 2026Yuwen Li, Guozhi ZhangNeural Network Approximation TheoryFunction Approximation

    School of Mathematical Sciences, Zhejiang University, 866 Yuhangtang Road, Hangzhou 310058, Zhejiang, China

  18. Every Layer Counts: An Exponential L2L_2 Depth Hierarchy for ReLU Networks

    Aug 24, 2026Itay SafranShallow Neural NetworksReLU Neural Networks

    Stein Faculty of Computer and Information Science, Ben-Gurion University of the Negev

  19. Resolution-Consistent Greedy Neural Approximation on Infinite-Dimensional Spaces

    Aug 21, 2026Pablo M. Berná, Antonio Falcó, Diego MondéjarNeural Network Approximation TheoryFunction Approximation

    Universidad Tecnol´ogica Atl´antico Mediterr´aneo – UTAMED, Facultad de Empresa Digital, Tecnolog´ıa y Derecho, M´alaga, Espa˜na · Departamento de Matem´aticas, F´ısica y Ciencias Tecnol´ogicas, Universidad Cardenal Herrera-CEU, CEU Universities, San Bartolom´e 55, Alfara del Patriarca (Valencia), 46115, Spain · Departamento de Matem´aticas, CUNEF Universidad, 28040, Espa˜na

  20. Convex losses and their applications to SVM, SVR, and Shallow Neural Networks

    Aug 14, 2026Filippo PorteraSupport Vector MachinesNeural Network Generalization

    DAIS, Università degli studi ”Ca’ Foscari” di Venezia, Italy

  21. A Novel Fourier Feature Network for Solving Partial Differential Equations

    Aug 13, 2026Qihong Yang, Zhijie Su, Yangtao Deng +1Shallow Neural NetworksFourier Feature Embeddings

    School of Mathematics, Sichuan University, city=Chengdu, postcode=610065, country=China

  22. Approximation Rates for Metaplectic Neural Networks

    Aug 9, 2026Ahmed Abdeljawad, Marcello Carioni, Elena CorderoShallow Neural NetworksNeural Network Approximation Theory

    Johann Radon Institute of Computational and Applied Mathematics (RICAM) Austrian Academy of Sciences Altenberger Straße 69, A-4040 Linz, Austria · Department of Applied Mathematics University of Twente 7500AE Enschede, The Netherlands · Dipartimento di Matematica Università degli Studi di Torino via Carlo Alberto 10, 10123 Torino, Italy

  23. A Counterexample to Fourier Alignment in Single-Neuron Modular Addition

    Aug 5, 2026Gautam Neelakantan MemanaShallow Neural NetworksReLU Neural Networks

  24. Shallower ReLU Network Representations via Exact Linear Algebra

    Jul 22, 2026Kilian Rueß, Gennadiy Averkov, Florestan Brunck +7Shallow Neural NetworksReLU Neural Networks

    University of Technology Nuremberg · Brandenburg University of Technology Cottbus–Senftenberg · University of Copenhagen +3

  25. Self-organizing Architecture of Receptron Units: a Hardware-Aware Framework for Edge Intelligence

    Jul 22, 2026Stefano Radice, Ludovico Casaccia, Riccaro Emanuele Beccalli +2Shallow Neural NetworksNeuromorphic Computing

    Department of Physics “Aldo Pontremoli” and CIMaINa University of Milano Milano, Italy

  26. Expressivity of Shallow Neural Networks Over Finite Fields

    Jul 19, 2026Maksym Zubkov, Carol Wu, Shiwei Yang +2Shallow Neural NetworksPolynomial Neural Networks

    Department of Mathematics, University of British Columbia, 1984 Mathematics Rd, Vancouver, V6T 1Z2, BC, Canada

  27. All you need is SAMPAT

    Jul 10, 2026Jayadeva, Madhur AswaniShallow Neural NetworksInterpretable ML

    Department of Electrical Engineering, Indian Institute of Technology, Delhi

  28. Implicit Bias of SGD in Multivariate ReLU Networks: Effective Width Collapse

    Jul 3, 2026Shuang Liang, Tom Jacobs, Guido MontúfarShallow Neural NetworksReLU Neural Networks

    Department of Statistics & Data Science University of California, Los Angeles Los Angeles, CA 90095, USA · CISPA Helmholtz Center for Information Security · Departments of Mathematics and Statistics & Data Science University of California, Los Angeles; and Max Planck Institute for Mathematics in the Sciences

  29. Born Discrete, Made Smooth: Variational Formulation of Shallow Neural Networks

    Jul 2, 2026Matej Benko, Pierre Bousquet, Iwona Chlebicka +1Shallow Neural NetworksNeural Network Generalization

    Institute of Mathematics, Faculty of Mechanical Engineering, Brno University of Technology · Technická 2896/2, 616 69, Brno, Czech Republic · Université de Toulouse, INSA Toulouse, CNRS, IMT, F-31062 Toulouse Cedex 9, France +2

  30. Neuro-Bayesian-Symbolic Residual Attention Shallow Network: Explainable Deep Learning for Cybersecurity Risk Assessment

    Jun 29, 2026Nicolaie Popescu-Bodorin, Madeleine TogherShallow Neural NetworksExplainable Artificial Intelligence

    Engineering & Computer Science Department Spiru Haret University, Bucharest, Romania · Computer Information Science Department Higher Colleges Technologies, Sharjah, U.A.E.