Neural Network Expressivity

Follow this topic Ask about this topic

Momentum

7 papers in the last four weeks, up 40% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 92

All topics

Identifying structural design principles shaping the computational abilities of recurrent neural networks
Jun 22, 2026Tom Talpir, Elad SchneidmanRecurrent Neural Networks Neural Network Expressivity
Department of Brain Sciences, Weizmann Institute of Science, Rehovot, Israel
EML Trees Are Universal Approximators
Jun 22, 2026Joe Germany, Elie Abdo, Joseph BakarjiUniversal Approximation Neural Network Approximation Theory
Department of Mathematics American University of Beirut Beirut, Lebanon · Department of Mechanical Engineering American University of Beirut Beirut, Lebanon
On the Expressive Power of Weight Quantization in Large Language Models
Jun 20, 2026Shao-Qun ZhangLLM Quantization Weight-Only Quantization
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210063, China. · School of Intelligent Science and Technology, Nanjing University, Suzhou 215163, China.
Expressivity Saturation: Reduced Affine Region Usage Under Increasing Task Complexity
Jun 19, 2026Xuan Qi, Yi Wei, Fanqi Yu +1Multilayer Perceptrons Neural Network Approximation Theory
AI for Good (AIGO), Istituto Italiano di Tecnologia · DITEN, University of Genoa · State Key Laboratory of Novel Software Technology +1
Effects of sparsity and superposition on loss in simple autoencoders
Jun 16, 2026Mriganka Basu Roy Chowdhury, Eric McLaughlin WeinerFeature Superposition Activation Sparsity
Department of Statistics, UC Berkeley · Department of Materials Science, UC Berkeley
Neuro-Relational Programs: Unifying Queries and Neural Computation over Structured Data
Jun 10, 2026Arie Soeteman, Balder ten Cate, Maurice Funk +3Graph Neural Networks Relational Reasoning
ILLC, University of Amsterdam, Amsterdam, Netherlands · Leipzig University, Leipzig, Germany · ScaDS.AI Center Dresden/Leipzig, Germany +2
Characterizing the Discrete Geometry of ReLU Networks
Jun 5, 2026Blake B. Gaines, Jinbo BiNeural Representation Geometry ReLU Neural Networks
Department of Computer Science University of Connecticut
Rethinking Neural Width for Alternating Current Optimal Power Flow Proxies
Jun 2, 2026Dhruvi Khandelwal, Anurag Basistha, Ayushi Jolotia +1Neural Surrogate Modeling Optimal Power Flow
Department of Electrical Engineering, National Institute of Technology Kurukshetra, India. · Indraprastha Institute of Information Technology Delhi, India. · Department of Electrical Engineering, Indian Institute of Technology Roorkee, India.
Expressivity of congruence-based architectures for DNNs on positive-definite matrices
Jun 1, 2026Antonin Oswald, Estelle MassartNeural Network Expressivity
ICTEAM, UCLouvain · Louvain-la-Neuve, Belgium
An Algebraic View of the Expressivity of Recurrent Language Models
Jun 1, 2026Franz Nowak, Ryan Cotterell, Reda BoumasmoudRecurrent Neural Networks Neural Network Expressivity
ETHZürich.
Revisiting Padded Transformer Expressivity: Which Architectural Choices Matter and Which Don't
May 28, 2026Anej Svete, William Merrill, Ryan Cotterell +1Transformer Expressivity Transformer
1. ETH Zürich · 2. Allen Institute for AI
Expressive Power of Floating-Point Neural Networks with Arbitrary Reduction Orders and Inexact Activation Implementations
May 27, 2026Yeachan Park, Geonho Hwang, Wonyeol Lee +1Neural Network Activation Functions Universal Approximation
More Expressive Feedforward Layers: Part I. Token-Adaptive Mixing of Activations
May 26, 2026Mingze Wang, Jinbo Wang, Yikuan Xia +2Transformer FFNs Transformer Expressivity
ByteDance Seed · Peking University
Random Neural Network Expressivity for Non-Linear Partial Differential Equations
May 24, 2026Muhammed Ali Mehmood, Lukas GononNeural PDE Solvers PDE Solving
Department of Mathematics Imperial College London UK · School of Computer Science University of St. Gallen Switzerland
Characterizing the Representational Capacity of Neural Processes
May 22, 2026Robin YoungNeural Processes Neural Network Expressivity
University of Cambridge
Preisach Attention: A Hysteretic Model of Sequential Memory
May 22, 2026Piotr FrydrychTransformer Efficient Attention
The Metrology and Biomedical Engineering Institute · Faculty of Mechatronics, Warsaw University of Technology
The Deterministic Horizon: Impossibility Results as Design Specifications for Trustworthy AI Systems
May 21, 2026Dongxin GuoTransformer Expressivity Trust in AI
UNIVERSITY OF HONG KONG
Holographic functions and neural networks
May 21, 2026Balazs SzegedyNeural Network Approximation Theory Neural Network Expressivity
Lost in Tokenization: Fundamental Trade-offs in Graph Tokenization for Transformers
May 21, 2026Maya Bechler-Speicher, Gilad Yehudai, Gil Harari +3Transformer Expressivity Graph Representation Learning
Meta AI · John A. Paulson School of Engineering and Applied Sciences, Harvard University · Google Research +2
How Many Different Outputs Can a Transformer Generate?
May 21, 2026Maxime Meyer, Mario Michelessa, Caroline Chaux +1Transformer Expressivity Transformer
Department of Mathematics, National University of Singapore, Singapore, 117543 · IPAL, IRL2955, Singapore · School of Computing, National University of Singapore, Singapore, 117543 +2
Approximation Theory for Neural Networks: Old and New
May 20, 2026Soumendu Sundar Mukherjee, Himasish TalukdarUniversal Approximation Neural Network Approximation Theory
Statistics and Mathematics Unit, Indian Statistical Institute, Kolkata
A Measure-Theoretic Analysis of Reasoning: Structural Generalization and Approximation Limits
May 19, 2026Yuyang Zhang, Yifu Zhang, Xuehai Zhou +1Transformer Expressivity OOD Generalization
McGill University · Mila - Quebec AI Institute · Université de Montréal
Updating the standard neuron model in artificial neural networks
May 19, 2026Raul Mohedano, Thomas Batard, Erik Velasco-Salido +4Neural Network Robustness Neural Network Expressivity
1Spanish National Research Council (CSIC), Madrid, Spain. · Center for Research in Mathematics (CIMAT), Guanajuato, Mexico. · 3Universidad Aut´onoma de Madrid (UAM), Madrid, Spain. +1
The Expressive Power of Low Precision Softmax Transformers with (Summarized) Chain-of-Thought
May 18, 2026Moritz Brösamle, Stephan EcksteinTransformer Expressivity Transformer
Department of Mathematics, University of Tübingen, Germany.
Beyond Accuracy: Robustness, Interpretability and Expressiveness of EEG Foundation Models
May 17, 2026Urban Širca, Maryam Alimardani, Stefanos Zafeiriou +1EEG Decoding Neural Network Interpretability
Vrije Universiteit Amsterdam · Imperial College London
The WidthWall: A Strict Expressivity Hierarchy for Hypergraph Neural Networks
May 13, 2026Fengqing Jiang, Yuetai Li, Yichen Feng +6GNN Expressivity Hypergraph Neural Networks
University of Washington · Western Washington University · King Abdulaziz City for Science and Technology +1
Lower bounds for one-layer transformers that compute parity
May 12, 2026Daniel HsuTransformer Expressivity Self-Attention
Scaling Laws and Tradeoffs in Recurrent Networks of Expressive Neurons
May 12, 2026Aaron Spieler, Georg Martius, Anna LevinaRecurrent Neural Networks Neural Scaling Laws
University of Tübingen, Germany · Max Planck Institute for Intelligent Systems, Tübingen, Germany · Max Planck Institute for Biological Cybernetics, Tübingen, Germany
The Polynomial Counting Capabilities of Message Passing Neural Networks
May 11, 2026Marco Sälzer, Pascal Bergsträßer, Anthony W. LinGraph Neural Networks Message Passing Neural Networks
1RPTU University Kaiserslautern-Landau · 2Max Planck Institute for Software Systems (MPI-SWS)
Minimal Filling Architectures of Polynomial Neural Networks: Counterexamples, Frontier Search, and Defects
May 10, 2026Kevin Dao, Jose Israel RodriguezPolynomial Neural Networks Neural Network Approximation Theory
Department of Mathematics, University of Wisconsin-Madison, Wisconsin, USA.

Jun 22, 2026, q-bio.NCJ/K move · Enter open · S save

Identifying structural design principles shaping the computational abilities of recurrent neural networks

Tom Talpir, Elad Schneidman

Department of Brain Sciences, Weizmann Institute of Science, Rehovot, Israel

Recurrent Neural Networks Neural Network Expressivity

Open paper AskPDF arXiv

Understanding how the architecture of neural networks shapes the computations they carry is a central challenge in neuroscience and machine learning. While specific circuit architectures have been linked to particular network computations and theoretical bounds on expressivity of broad classes of networks have been found, we are still missing general principles connecting the structure of finite networks to their computational capabilities. Here, we characterize the computational abilities of recurrent neural networks as a function of their connectivity by training a large collection of different networks to compute a large set of Boolean functions. For small networks, we constructed the complete ``catalogs'' of network-function performance, which revealed that computational capacity varies widely across architectures and that most networks show poor performance, and most functions are hard to compute. However, we show that having local 2- and 3-cycles in a network strongly enhances its computational ability, and networks with such cycles are often the minimal architectures that can solve particular functions. We further show that a small set of structural statistics accurately predict networks' performance. Extending our analysis to large networks showed that typical networks fail even to approximate a randomly selected function. Surprisingly, adding a small number of sparsely connected biologically-inspired interneurons to the network dramatically increases computational capacity. As in small networks, adding short cycles improved networks' capacity, outperforming acyclic or reachability-matched controls. Thus, our results identify local cycles as design principles linking neural connectivity to computational power, and offer a general framework to explore structure-function relations in computing networks.

Momentum

Related topics

Latest papers 92