Neural Network Expressivity

Follow this topic Ask about this topic

It asks what mappings a model can represent exactly or approximate, and how the needed width, depth, parameters, precision, and activation choices scale with the task. Common approaches include universal-approximation results and lower bounds, counting linear regions or polynomial features, circuit-complexity and logic characterizations, and geometric analyses of MLPs, recurrent models, transformers, and graph networks. These tools reveal when depth, recurrence, multiplicative gates, or higher-order interactions add capabilities, often using tasks such as parity, memorization, graph distinguishing, and smooth-function approximation. A central tension is that theoretical capacity does not ensure training will find or use it, and results under idealized assumptions about precision or weights may not transfer directly to practical models.

Latest papers

118 papers8 papers in the last four weeks, down 11% on the four weeks before. 0.0% of all new papers.

The Geometry of Limits in ReLU Function Spaces
Oct 24, 2026Kathryn A. LindseyReLU Neural Networks Neural Network Approximation Theory
Internalizing Chain-of-thought into Looped Transformers
Oct 19, 2026Kehang Zhou, Pinyan LuRecurrent Transformers Latent Reasoning
Institute for Theoretical Computer Science Shanghai University of Finance and Economics
Upper bounds on the number of causal pieces in spiking neural networks
Oct 15, 2026Dominik Dold, Dennis Elbrächter, Philipp PetersenSpiking Neural Networks Neural Network Expressivity
The minimal width of neural networks and immersions of real projective space
Oct 14, 2026BaoZhi Chu, Ben WeinkoveNeural Network Approximation Theory Neural Network Expressivity
The Surprising Expressivity of a Single Neuron with Autoregressive Chain-of-Thought
Oct 12, 2026Roey Magen, Nirmit Joshi, Gal VardiNeural Network Expressivity CoT Reasoning
Weizmann Institute of Science · Toyota Technological Institute at Chicago
Constructive Bounds for the Topological Expressivity of Neural Networks
Oct 9, 2026Kosio BeshkovNeural Network Expressivity ReLU Neural Networks
Department of Physics, University of Oslo, Oslo, Norway.
A Hand-Checkable Proof That Two Hidden ReLU Layers Compute the Maximum of Six Numbers
Oct 3, 2026Dimitrios MyrisiotisReLU Neural Networks Neural Network Expressivity
Great Bay University China
Arbitrary-Accuracy Neural Approximation with Optimal Neuron Count and Near-Optimal Bit Complexity
Sep 28, 2026Zilan Cheng, Li-Lian Wang, Zhongjian WangNeural Network Approximation Theory Feedforward Neural Networks
Division of Mathematical Sciences, School of Physical and Mathematical Sciences, Nanyang Technological University, 637371, Singapore.
The Computational Value of Sensory-Aligned Receptive Fields Depends on Neuronal Expressivity
Sep 22, 2026Agnese Adorante, Aaron Spieler, Anna LevinaRecurrent Neural Networks Neural Network Expressivity
Department of Computer Science, University of Tübingen, Tübingen, Germany · Max Planck Institute for Biological Cybernetics, Tübingen, Germany
Neural Approximation by Function Composition: Rigidity and Doubly Exponential Convergence
Sep 22, 2026Wentao Huang, Haizhang ZhangNeural Network Approximation Theory Neural Network Expressivity
Department of Mathematics, The University of Hong Kong, Hong Kong, P.R. China. · School of Mathematics, Sun Yat-sen University, Guangzhou, P.R. China.
Graded Representation Theory of Equivariant Neural Networks
Sep 22, 2026Mani ShayestehfarEquivariant Neural Networks Equivariant Representation Learning
Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention
Sep 21, 2026Julien Siems, Riccardo Grazzi, Korbinian Pöppel +8Length Generalization Delta-Rule Linear Attention
♡University of Freiburg · ♢Microsoft Research · ELLIS Institute Tübingen +9
Algebraic Expressivity Certificates for Shallow Polynomial Neural Networks
Sep 11, 2026Sepehr Akbari, Shahrzad JamshidiNeural Network Expressivity Shallow Neural Networks
Dept. of Math & Computer Science Lake Forest College
Per-Node Activation Function Evolution in Indirectly Encoded Substrates: Solvability, Limits, and Emergent Diversity
Sep 10, 2026Romain Claret, Michael O'Neill, Paul Cotofrei +1Neural Network Activation Functions Neuroevolution
University of Neuchâtel, Switzerland · University College Dublin, Ireland
Polyhedral Geometry of Time-to-First-Spike Neural Networks
Sep 10, 2026Manjot Singh, Guido Montúfar, Gitta KutyniokSpiking Neural Networks Feedforward Neural Networks
Department of Mathematics, Ludwig-Maximilians-Universität München · Munich Center for Machine Learning · Departments of Mathematics and Statistics & Data Science, University of California, Los Angeles +2
Diversity of EML-type operators
Sep 10, 2026Andrzej OdrzywołekNeural Network Activation Functions Neural Network Expressivity
Institute of Theoretical Physics, Jagiellonian University, Łojasiewicza 11, 30-348 Kraków, Poland
Sharp Approximation Rates for Neural Networks with Affine Latent Parameterizations
Aug 31, 2026Shijun ZhangHypernetworks Neural Network Approximation Theory
Department of Applied Mathematics Hong Kong Polytechnic University
Representing MAX functions using two-hidden-layer ReLU networks
Aug 25, 2026Zhimao Wang, Amitabh BasuReLU Neural Networks Neural Network Expressivity
Department of Applied Mathematics, Johns Hopkins University
Parameter-Level Attribution of Symmetry in Trained Networks Though Parameter-Wise Functional Sensitivity
Aug 25, 2026Alan Muriithi, Vedanta Thapar, Torben BerndtEquivariant Neural Networks Neural Network Interpretability
Mathematical Institute, University of Oxford · Heidelberg Institute for Theoretical Studies
Every Layer Counts: An Exponential $L_2$ Depth Hierarchy for ReLU Networks
Aug 24, 2026Itay SafranShallow Neural Networks ReLU Neural Networks
Stein Faculty of Computer and Information Science, Ben-Gurion University of the Negev
Expressivity In Multimodal Contrastive Learning
Aug 17, 2026Andrew Stuart, Florian WolfMultimodal Contrastive Learning Neural Network Expressivity
California Institute of Technology
The Boolean Power of ReLU
Aug 12, 2026Pablo Barceló, Floris Geerts, Matthias Lanzinger +2ReLU Neural Networks GNN Expressivity
Pontifical Catholic University · IMFD & CENIA, Chile · University of Antwerp +4
Reducing Symmetry Increase in Equivariant Neural Networks
Aug 12, 2026Ning Lin, Jiacheng Cen, Anyi Li +2Equivariant Neural Networks Equivariant Representation Learning
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China
Three Tokens Force Exponential Feature Rank in Nonnegative Kernel Attention
Aug 11, 2026Vicente OpazoLow-Rank Attention Attention Mechanisms
CENIA
The Spectral Neuron
Aug 8, 2026Alex ShtoffNeural Network Robustness Neural Network Interpretability
Technology Innovation Institute
Neural network realization of binary refinement iterates via a two-chart atlas selector
Jul 21, 2026Tsogtgerel GantumurReLU Neural Networks Neural Network Approximation Theory
McGill University, Montr´eal, QC, Canada · National University of Mongolia, Ulaanbaatar, Mongolia · Mongolian Academy of Sciences, Institute of Mathematics and Digital Technology
Spectral Higher-Order Neural Networks Have Sharp Expressivity Bounds
Jul 21, 2026Gianluca Peri, Diego Febbe, Duccio FanelliHypergraph Neural Networks Neural Network Expressivity
Department of Physics, University of Florence & INFN, Italy
Beyond the Edge of Chaos: Stability-Expressivity Transfer in Reservoir Forecasting
Jul 20, 2026Yao Du, Xingang WangLyapunov Stability Reservoir Computing
School of Physics and Information Technology, Shaanxi Normal University, Xi’an 710062, China
Expressivity of Shallow Neural Networks Over Finite Fields
Jul 19, 2026Maksym Zubkov, Carol Wu, Shiwei Yang +2Shallow Neural Networks Polynomial Neural Networks
Department of Mathematics, University of British Columbia, 1984 Mathematics Rd, Vancouver, V6T 1Z2, BC, Canada
Algebraic Representability as the Limiting Regime of Grokking: An Exactly Solvable Model with Holomorphic Activations
Jul 15, 2026Chon-Fai Kam, Xavier Cadet, Miloud Bessafi +1Feedforward Neural Networks Neural Network Expressivity
University Paris City & University of Reunion, Paris, France · Dipartimento di Fisica e Chimica Emilio Segrè, Università degli Studi di Palermo, Palermo, Italy · Thayer School of Engineering, Dartmouth College, Hanover, NH 03755, USA +2

Oct 24, 2026J/K move · Enter open · S save

The Geometry of Limits in ReLU Function Spaces

Kathryn A. Lindsey

ReLU Neural Networks Neural Network Approximation Theory Neural Network Expressivity Shallow Neural Networks

Open paper AskPDF arXiv

We study the geometry and closure of spaces of functions realized by ReLU neural networks of fixed architecture. We first establish a general result about limits of continuous piecewise-linear (CPWL) functions: the minimum number of convex sets on which a function is affine is lower semicontinuous under pointwise convergence. The proof uses a connection to hypergraph coloring and a compactness principle, and yields quantitative bounds on the polyhedral complexity of continuous limits. We then characterize the functions realizable by shallow ReLU networks with vector-valued outputs, extending a known scalar-valued characterization. Using this and the geometry of limiting hyperplane arrangements, we prove that fixed-width shallow scalar ReLU realization spaces are closed in the weighted supremum norm on the whole input space. In contrast, we construct an explicit sequence of networks with architecture

(2,3,2\mid1)

whose limit is not realizable by that architecture. Non-closedness holds both globally in weighted supremum norm and under uniform convergence on compact sets with nonempty interior, disproving a conjecture of Petersen, Raslan, and Voigtlaender. Finally, we show that non-closedness for a fixed scalar-output ReLU architecture necessarily involves a blow-up residue, in which diverging output coefficients amplify an emerging linear dependence among hidden features. Together, these results provide geometric and functional-analytic tools for studying the boundaries and degenerations of ReLU realization spaces.